Got it

HBase Full Migration Procedure

Latest reply: May 9, 2022 12:21:04 436 1 1 0 0

1. Migration Procedure

The procedure for displaying the page is as follows:

1. Stop the HBase service in the source cluster.

2. Stop the HBase service of the target cluster.

3. HBase data migration

4. Check the ACL of the HBase directory.

5. HBase data size verification

6. Start the HBase service of the target cluster.

7. HBase metadata restoration

8. Start the HBase service of the source cluster.

2. Detailed steps in the HBase full migration scenario

2.1. Stopping and Starting the HBase Service

The detailed steps are omitted.

2.2. Step 3 Data Migration

Check whether all data in the source cluster is stored to disks. In the source cluster, check whether the /hbase/WALs directory in the HDFS directory is empty. If the directory is not empty, start HBase and stop HBase services.


hdfs dfs -count hdfs://${source_cluster_ip}/hbase/WALs # Client check on source cluster must return 0

Check whether the target cluster is the initial HBase service. If not, delete the HBase service and reinstall it.


hdfs dfs -count hdfs://${target_cluster_ip}/hbase/data/default # The return value of the client operation in the target cluster must be 0.

hdfs dfs -ls hdfs://${target_cluster_ip}/hbase/data/ | wc -l # The return value of the client operation in the target cluster must be 3.

Run the distcp command to migrate data.


hadoop distcp -Dmapreduce.job.queuename="${queue_name}" -strategy dynamic -bandwidth ${bandwidth} -m ${mr_num} -update -delete -append -prbugpcaxt hdfs://haclusterX/hbase/data hdfs://hacluster/hbase/data >> ${distcp_log} 2>&1

2.3 Step 4: Check the ACL.

Run the following command to obtain ACL information:

hadoop fs -getfacl hdfs://haclusterX/hbase/data >> ${source_acl_info}

hadoop fs -getfacl hdfs://hacluster/hbase/data >> ${target_acl_info}

Compare the ACL information on the two sides. If the ACL information is different, run the following command to modify the ACL information:

hadoop fs -setfacl --set ${acl_info} hdfs://hacluster/hbase/data

2.4. Step 5 Data Size Verification

2.4.1 Run the following command to obtain the file size information:

hadoop fs -stat -R "%F,%b,%o,%n" hdfs://haclusterX/hbase/data > ${source_data_info}

hadoop fs -stat -R "%F,%b,%o,%n" hdfs://hacluster/hbase/data > ${target_data_info}

2.4.2 Check whether the two data_info files are different. If they are different, the migration fails. Run the distcp command again.

2.4.3 Run zkCli.sh to delete HBase data stored in ZooKeeper in the target cluster.

The reason for performing this step is that there is an additional hindex directory in HBase 6.5.1. This directory is automatically created when the HBase service is started. However, if the ZooKeeper contains hindex information, the directory is not created.

kinit hbase #The HBase user must be used for authentication.

Command line for connecting the zkCli.sh -server ${zkhost}:24002 # to the ZooKeeper client

After entering the command line

deleteall /hbase

2.5. Step 6 Restoring HBase Metadata

After the HBase service of the target cluster is started, run the following command to rectify the fault:

kinit hbase #The HBase user must be used for authentication.

hbase hbck -repair # Run the command repeatedly until "Status OK" is displayed.


  • x
  • convention:

wissal
MVE Created May 9, 2022 12:21:04

Learned, keep up the good work!
View more
  • x
  • convention:

Comment

You need to log in to comment to the post Login | Register
Comment

Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:
  • Politically sensitive content
  • Content concerning pornography, gambling, and drug abuse
  • Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy
Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " User Agreement."

My Followers

Login and enjoy all the member benefits

Login

Block
Are you sure to block this user?
Users on your blacklist cannot comment on your post,cannot mention you, cannot send you private messages.
Reminder
Please bind your phone number to obtain invitation bonus.
Information Protection Guide
Thanks for using Huawei Enterprise Support Community! We will help you learn how we collect, use, store and share your personal information and the rights you have in accordance with Privacy Policy and User Agreement.