Got it

How to define the data row?

Latest reply: Dec 22, 2021 12:10:43 3049 5 13 0 0

Hello there, dear friends!

This post explains the procedure of how to define the data row. Please have a look at the information displayed below.

PROCEDURE

1. Put the data into the HDFS.

hdfs dfs -mkdir <inputdir> 

hdfs dfs -put <local_data_file> <inputdir> 

2. Create and define configuration.xml.

3. Details about how to define inapplicable data rows are as follows:

114540v53fojgyaimetdza.png


4. Run the following command to make the settings of configuration.xml take effect:

$ bin/hbase com.huawei.hadoop.hbase.tools.bulkload.ImportData

- Dimport.skip.bad.lines=false

- Dimport.bad.lines.output=</path/badlines/output>

- Dimport.hfile.output=</path/for/output> <configuration xmlfile> <tablename> <inputdir>

Dimport.skip.bad.lines: If this parameter is set to false, the command will stop at an inapplicable row. If this parameter is set to true, the command will skip the inapplicable row and continue to run.

Dimport.bad.lines.output=</path/badlines/output>: indicates the output path of the inapplicable data rows.

Dimport.hfile.output=< /path/for/output>: indicates the output path of the execution results.

<configuration xmlfile>: indicates the configuration file.

<tablename>: indicates the name of the table to be operated.

<inputdir>: indicates the directory where data is uploaded in batches.

5. Execute the command to import HFile to HBase. There are two ways to import:

Import data without the secondary index.

./hbase org.apache.hadoop.hbase.index.mapreduce.LoadIncrementalHFiles </path/for/output> <tablename> 

Import the data with the secondary index. We need to ensure that Region split will not happen during the process of bulkload.

If the data needed to import exceeds the default max size of one region, region split will occur. So we can set the max region size by setting the value of the hbase.hregion.max.filesize paramter in FusionInsight Manager.

./hbase org.apache.hadoop.hbase.index.mapreduce.IndexLoadIncrementalHFiles </path/for/output> <tablename>

This is the procedure of how to define the data row.

  • x
  • convention:

dagui
Created Dec 25, 2018 03:47:10

If the data needed to import exceeds the default max size of one region, region split will come. Can you provide a more detailed explanation?
View more
  • x
  • convention:

YOO
Created Dec 25, 2018 06:13:37

If the data needed to import exceeds the default max size of one region, region split will come.it is very helpful on How to define the data row
View more
  • x
  • convention:

yiyi0519
Created Dec 25, 2018 07:00:23

If you can add more figure, With these screenshots, the whole step is much clearer and easier to understand and learn.
View more
  • x
  • convention:

Mysterious.color
Created Dec 25, 2018 13:53:47

thanks i know now How to define the data row
View more
  • x
  • convention:

olive.zhao
Admin Created Dec 22, 2021 12:10:43

Thanks for your sharing!
View more
  • x
  • convention:

Comment

You need to log in to comment to the post Login | Register
Comment

Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:
  • Politically sensitive content
  • Content concerning pornography, gambling, and drug abuse
  • Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy
Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " User Agreement."

My Followers

Login and enjoy all the member benefits

Login

Block
Are you sure to block this user?
Users on your blacklist cannot comment on your post,cannot mention you, cannot send you private messages.
Reminder
Please bind your phone number to obtain invitation bonus.
Information Protection Guide
Thanks for using Huawei Enterprise Support Community! We will help you learn how we collect, use, store and share your personal information and the rights you have in accordance with Privacy Policy and User Agreement.