HBase SystemTable namespace Is Deleted Due to Misoperation

Fault locating

Stop the HBase service and delete the related namespace data from ZooKeeper. Then, restore the HBase table path and description information in HDFS, start the HBase service and manually restore the data.

1. Stop the HBase service.

http://10.88.194.32:7088/idp-edit-service/editor/image/11108756775/A-1_1_en-us_image_0147205341.jpg

2. Run the following command to go to the client installation directory, for example, /opt/client:

3. Run the source bigdata_env command to configure environment variables.

4. Perform the security authentication if the cluster uses a secure version.

NOTE:

You need to perform the authentication as the hbase user for all the following steps.

1. Enter the correct password as prompted.

[root@10-10-10-11 client]# source bigdata_env  #configure environment variables

[root@10-10-10-11 client]# kinit hbase         #set kinit authentication

2. Password for hbase@HADOOP.COM:

Log in to the ZooKeeper client hbase zkcli or zkCli.sh -server zkip:24002 and delete all namespace tables created by users.For example:

[root@10-10-10-11 client] # hbase zkcli

[zk: hadoop01:24002,hadoop02:24002,hadoop03:24002(CONNECTED) 0] ls /hbase/namespace

[default, hbase, lgp, lgp2]

NOTE:

1. In the preceding information, default and hbase are the two default namespace databases. lgp and lgp2 are the databases created by users.

2. Recreate the description file path of the HBase namespace table in HDFS. For example:

3. Restore the namespace table description file .tableinfo.0000000001 under /hbase/data/hbase/namespace/.tabledesc.

4. Copy the table description file from another cluster of the same version to the file directory as required. Pay attention to the permission. For example:

-rw------- 3 hbase supergroup        312 2018-02-05 01:42 /hbase/data/hbase/namespace/.tabledesc/.tableinfo.0000000001

5. Start the HBase service.

http://10.88.194.32:7088/idp-edit-service/editor/image/11108756775/A-1_1_en-us_image_0147205343.png

6. Back up the data of all tables created by users in the namespace database. For example:

7. Run the hbase shell command to log in to the HBase client and recreate all namespace databases that have been deleted in Step 5. For example:

hbase(main):031:0> create_namespace 'lgp'

2018-02-05 01:42,276 INFO[main] ipc.AbstractRpcClient: RPC Server Kerberos principal name for service=MasterService is hbase/hadoop.hadoop.com@HADOOP.COM

0 row(s) in 0.1840 seconds

8. Restore the namespace tables that have been backed up in Step 9. For example:

[root@10-10-10-11 client] # dfs dfs -mv /hbase/data/lgp_bak/* /hbase/data/lgp/

NOTE:

1. Perform similar operations for other namespace tables.

2. Run the hbase hbck -fixMeta command to restore the meta table.

3. If the restoration is successful, OK is displayed inthe command output. For example:

[root@10-10-10-11 client] # hbase hbck -fixMeta

http://10.88.194.32:7088/idp-edit-service/editor/image/11108756775/A-1_1_en-us_image_0147205345.jpg

1. Run the hbase shell command to log in to the HBase client and run the list command and check whether the created user tables are displayed.

For example:

hbase(main):003:0> list

TABLE 2018-02-05 01:42,649 INFO  [main] ipc.AbstractRpcClient: RPC Server Kerberos principal name for service=MasterService is hbase/hadoop.hadoop.com@HADOOP.COM

JH_TAB                                                             

_ctmeta_                                                                                 
big_region_tab                                                                              
big_region_tab_2                                                                           
big_region_tab_3                                                                        
big_region_tab_4                                                                           
filter_test                                                                              
hbase_sample_table                                                                     
hive_hbase_student_3                                                                       
lgp2:t1 lgp2:t2

lgp:t1 lgp:t2                                                                                            
stu_ctbase_table                                                                        
test_tab                                                                                 
test_tab_idx                                                                           
16 row(s) in 0.0480 seconds

=> ["JH_TAB", "_ctmeta_", "big_region_tab", "big_region_tab_2", "big_region_tab_3", "big_region_tab_4", "filter_test", "hbase_sample_table", "hive_hbase_student_3", "lgp2:t1", "lgp2:t2", "lgp:t1", "lgp:t2", "stu_ctbase_table", "test_tab", "test_tab_idx"]

2. The HBase status is normal.

http://10.88.69.41:7088/idp-edit-service/editor/image/11108756775/A-1_1_en-us_image_0147205347.jpg