Hi there!
Fault handling
Rebuild the meta table using the HDFS data that maps to HBase.
Solution
1. Stop the HBase service.
http://10.88.69.41:7088/idp-edit-service/editor/image/11108756775/A-1_1_en-us_image_0084228388.jpg
2. Run the following command to go to the client installation directory, for example, /opt/client:
cd /opt/client
3. Run the source bigdata_env command to configure the environment variables.
4. If the cluster uses the security version, perform security authentication.
kinit hbase
5. Enter the correct password as prompted.
[root@10-10-10-11 client]# source bigdata_env #Configure environment variables.
[root@10-10-10-11 client]# kinit hbase #Set kinit authentication.
Password for hbase@HADOOP.COM.
6. Run the HDFS command hdfs dfs -ls /hbase/data to obtain HBase table information. It is found that only the default and hbase directories exist in the command output. Example:
root@10-10-10-11 client] # hdfs dfs -ls /hbase/data
drwxr-xr-x - hbase hadoop 0 2016-12-15 16:37 /hbase/data/default
drwxr-xr-x - hbase hadoop 0 2016-12-15 16:37 /hbase/data/hbase
7. Run the HDFS command hdfs dfs -ls /user to obtain directory information. Example:
[root@10-10-10-11 client] # hdfs dfs -ls /user
Found 11 items
drwx------ - admin hadoop 0 2016-12-15 16:37 /user/admin
drwxr-xr-x - hbase hadoop 0 2016-12-15 16:37 /user/hbase
drwxrwx--- - hive hive 0 2016-12-15 16:37 /user/hive
drwx------ - admin hadoop 0 2016-12-15 16:37 /user/impala
drwxrwxrwx - loader hadoop 0 2016-12-15 16:37 /user/loader
drwxr-xr-x - mapred hadoop 0 2016-12-15 16:37 /user/mapred
drwx--x--x - hbase hadoop 0 2016-12-15 16:37 /user/omm
drwxr-xr-x - spark hadoop 0 2016-12-15 16:37 /user/spark
drwxr-xr-x - spark hadoop 0 2016-12-15 16:37 /user/sparkhive
drwx------ - test hadoop 0 2016-12-15 16:37 /user/test
If the hbase directory exists, run the hdfs dfs -mv command to rename the directory.
[root@10-10-10-11 client] # hdfs dfs -mv /user/hbase /user/hbase_bak
8. Run the HDFS commands to migrate the namespace and meta information in the hbase directory.
hdfs dfs -mkdir /user/hbase
hdfs dfs -mv /hbase/data/hbase/namespace /user/hbase/namespace
hdfs dfs -mv /hbase/data/hbase/meta /user/hbase/meta
9. Run the following HDFS command:
[root@10-10-10-11 client] # hdfs dfs -ls /hbase/data/hbase
drwxr-xr-x - hbase supergroup 0 2016-12-15 16:37 /hbase/data/hbase/acl
#This directory exists in a security cluster.
10. Run the following ZooKeeper command to delete the Znode:
zkCli.sh -server zk_host:port/chroot
Note:
Replication and hbaseid cannot be deleted.
[root@10-10-10-11 client]# zkCli.sh -server 10.149.0.90:24002 #Enter
Connecting to
10.149.0.90:24002
Welcome to ZooKeeper!
JLine support is enabled
[zk:
10.149.0.90:24002(CONNECTING) 0]
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
WATCHER::
WatchedEvent state:SaslAuthenticated type:None path:null
[zk:
10.149.0.90:24002(CONNECTED) 0]
ls /hbase[acl, backup-masters, balancer, draining, flush-table-proc, hbaseid, master, meta-region-server, namespace, online-snapshot, recovering-regions, region-in-transition, replication, rs, running, splitWAL, table, table-lock, tokenauth]
[zk:
10.149.0.90:24002(CONNECTED) 1]
deleteall /hbase/acl #delete
......
[zk:
10.149.0.90:24002(CONNECTED) 20]
quit
11. Start the HBase service.
Note:
The start fails if no metadata exists.
http://10.88.194.32:7088/idp-edit-service/editor/image/11108756775/A-1_1_en-us_image_0084228390.jpg
Run the HBase command to repair metadata:
hbase hbck -repair
Example:
[root@10-10-10-11 client]# hbase hbck -repair #Repair
HBaseFsck command line options: -repair
Version: 1.0.2
Number of live region servers: 3
Number of dead region servers: 0
Master: 8-5-144-5,21300,1488786342838
Number of backup masters: 1
Average load: 0.6666666666666666
Number of requests: 1
Number of regions: 2
Number of regions in transition: 0
Number of empty REGIONINFO_QUALIFIER rows in hbase:meta: 0
Number of Tables: 1
.
ERROR: Region { meta => null, hdfs => hdfs://hacluster/hbase/data/default/table9/f0aa77d6ddfcd031f7550e2f9f79a513, deployed => , replicaId => 0 } on HDFS, but not listed in hbase:meta or deployed on any region server
Trying to reassign region...
Trying to reassign region...
Trying to reassign region...
Trying to reassign region...
Trying to reassign region...
Trying to reassign region...
Trying to reassign region...
Trying to reassign region...
Trying to reassign region...
Trying to reassign region...
Trying to reassign region...
Trying to reassign region...
0 inconsistencies detected.
Status: OK
