Got it

HBase BulkLoad Task Failure with Bulk Load in FusionInsight

Created: Dec 7, 2021 09:06:14Latest reply: Jan 13, 2022 11:40:15 373 5 0 0 0
  Rewarded HiCoins: 0 (problem resolved)

Hi


Why does an HBase BulkLoad task fail if it consists of 210,000 maps and 10,000 reductions (a single table contains 26T data) in FusionInsight?
  • x
  • convention:

Featured Answers
SalemKhan
Created Jan 13, 2022 11:40:15

ncrease the value of the RPC timeout parameter of the client.

cd ${clientPah}/HBase/hbase/conf

vi hbase-site.xml

Locate hbase.rpc.timeout, increase the value, save, and exit.
icon-note.png NOTE:

clientPah indicates the client installation root directory specified by a customer. Replace clientPah with your actual installation directory.


Method to identify a ZooKeeper IO bottleneck


1. On the Manager monitoring page, view the monitoring result of the ZooKeeper request for a single node to determine if the specified limit is exceeded.

2. Check the ZooKeeper log and the HBase log for a large number of I / O exception time-out errors or SocketTimeout time-out exception errors.


Recommendations for optimization:


1. Increase the number of ZooKeeper instances to 5 or more. It is recommended to increase the number of observers by setting peerType = observer.

2. Restrict the number of specific concurrent maps to a single task or reduce memory for each task running on the node to reduce the load on the node.

3. Update the ZooKeeper data disk, such as the SSD.
View more
  • x
  • convention:

AymanOT
MVE Author Created Dec 7, 2021 09:11:47

Hi


Possible Causes

Loading a large number of files triggers Compaction. As a result, data loading times out.

Heavy workload on HDFS causes timeout.


Solution

Increase the value of the RPC timeout parameter of the client.

cd ${clientPah}/HBase/hbase/conf

vi hbase-site.xml

Locate hbase.rpc.timeout, increase the value, save, and exit.

icon-note.png NOTE:

clientPah indicates the client installation root directory specified by a customer. Replace clientPah with your actual installation directory.


Method to identify a ZooKeeper IO bottleneck



1. On the Manager monitoring page, view the monitoring result of the ZooKeeper request for a single node to determine if the specified limit is exceeded.

2. Check the ZooKeeper log and the HBase log for a large number of I / O exception time-out errors or SocketTimeout time-out exception errors.


Recommendations for optimization:


1. Increase the number of ZooKeeper instances to 5 or more. It is recommended to increase the number of observers by setting peerType = observer.

2. Restrict the number of specific concurrent maps to a single task or reduce memory for each task running on the node to reduce the load on the node.

3. Update the ZooKeeper data disk, such as the SSD.
View more
  • x
  • convention:

All Answers
little_fish
little_fish Admin Created Dec 7, 2021 09:08:48

Dear Ms.Sar95,
What's the version of FI and HBase? What‘s the error code? Thanks.

View more
  • x
  • convention:

Ms.Sar95
Ms.Sar95 Created Dec 7, 2021 09:13:57 (0) (0)
"Attempt to bulk load region failed"

V100R002C60U10  
AymanOT
AymanOT MVE Author Created Dec 7, 2021 09:11:47

Hi


Possible Causes

Loading a large number of files triggers Compaction. As a result, data loading times out.

Heavy workload on HDFS causes timeout.


Solution

Increase the value of the RPC timeout parameter of the client.

cd ${clientPah}/HBase/hbase/conf

vi hbase-site.xml

Locate hbase.rpc.timeout, increase the value, save, and exit.

icon-note.png NOTE:

clientPah indicates the client installation root directory specified by a customer. Replace clientPah with your actual installation directory.


Method to identify a ZooKeeper IO bottleneck



1. On the Manager monitoring page, view the monitoring result of the ZooKeeper request for a single node to determine if the specified limit is exceeded.

2. Check the ZooKeeper log and the HBase log for a large number of I / O exception time-out errors or SocketTimeout time-out exception errors.


Recommendations for optimization:


1. Increase the number of ZooKeeper instances to 5 or more. It is recommended to increase the number of observers by setting peerType = observer.

2. Restrict the number of specific concurrent maps to a single task or reduce memory for each task running on the node to reduce the load on the node.

3. Update the ZooKeeper data disk, such as the SSD.
View more
  • x
  • convention:

Unicef
Unicef MVE Created Dec 8, 2021 12:28:55

GOOD ANSWER
View more
  • x
  • convention:

SalemKhan
SalemKhan Created Jan 13, 2022 11:40:15

ncrease the value of the RPC timeout parameter of the client.

cd ${clientPah}/HBase/hbase/conf

vi hbase-site.xml

Locate hbase.rpc.timeout, increase the value, save, and exit.
icon-note.png NOTE:

clientPah indicates the client installation root directory specified by a customer. Replace clientPah with your actual installation directory.


Method to identify a ZooKeeper IO bottleneck


1. On the Manager monitoring page, view the monitoring result of the ZooKeeper request for a single node to determine if the specified limit is exceeded.

2. Check the ZooKeeper log and the HBase log for a large number of I / O exception time-out errors or SocketTimeout time-out exception errors.


Recommendations for optimization:


1. Increase the number of ZooKeeper instances to 5 or more. It is recommended to increase the number of observers by setting peerType = observer.

2. Restrict the number of specific concurrent maps to a single task or reduce memory for each task running on the node to reduce the load on the node.

3. Update the ZooKeeper data disk, such as the SSD.
View more
  • x
  • convention:

Comment

You need to log in to comment to the post Login | Register
Comment

Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:
  • Politically sensitive content
  • Content concerning pornography, gambling, and drug abuse
  • Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy
Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " User Agreement."

My Followers

Login and enjoy all the member benefits

Login

Block
Are you sure to block this user?
Users on your blacklist cannot comment on your post,cannot mention you, cannot send you private messages.
Reminder
Please bind your phone number to obtain invitation bonus.
Information Protection Guide
Thanks for using Huawei Enterprise Support Community! We will help you learn how we collect, use, store and share your personal information and the rights you have in accordance with Privacy Policy and User Agreement.