Got it
Huawei Enterprise Support Community
Community Forums Cloud & Big Data
The Service Becomes Abnor...

The Service Becomes Abnormal After Description Information in the Metadata Namespace Table Is Lost

Created: Jul 25, 2019 06:50:16Latest reply: Jul 25, 2019 06:50:59 392 1 1 0 1
  Rewarded HiCoins： 0 (problem resolved)
View the author 1#

The Service Becomes Abnormal After Description Information in the Metadata Namespace Table Is Lost.

Symptom

The HBase fails to be started.

http://10.88.194.32:7088/idp-edit-service/editor/image/11108756775/A-1_1_en-us_image_0119804016.png

The HMaster log shows that the region in the hbase:namespace table fails to go online.

http://10.88.194.32:7088/idp-edit-service/editor/image/11108756775/A-1_1_en-us_image_0117892721.png

The RegionServer log contains the following error information:

2018-03-14 05:35,313 | INFO | PriorityRpcServer.handler=2,queue=0,port=21302 | Open hbase:namespace,,1520587645018.307a208051bc027972c1ed66ead8dc06. | org.apache.hadoop.hbase.regionserver.RSRpcServices.openRegion(RSRpcServices.java:1517)
2018-03-14 05:35,332 | ERROR | RS_OPEN_REGION-pzhisfeapsvr005:21302-2 | Failed open of region=hbase:namespace,,1520587645018.307a208051bc027972c1ed66ead8dc06., starting to roll back the global memstore size. | org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:386)
java.lang.IllegalStateException: Could not instantiate a region instance.
at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6317)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6624)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6596)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6552)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6503)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:363)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:129)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedConstructorAccessor37.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6314)
... 10 more
Caused by: java.lang.IllegalArgumentException: Need table descriptor
at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:715)
at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:692)
... 14 more
2018-03-14 05:35,332 | INFO | RS_OPEN_REGION-pzhisfeapsvr005:21302-2 | Opening of region {ENCODED => 307a208051bc027972c1ed66ead8dc06, NAME => 'hbase:namespace,,1520587645018.307a208051bc027972c1ed66ead8dc06.', STARTKEY => '', ENDKEY => ''} failed, transitioning from OPENING to FAILED_OPEN in ZK, expecting version 4261 | org.apache.hadoop.hbase.coordination.ZkOpenRegionCoordination.tryTransitionFromOpeningToFailedOpen(ZkOpenRegionCoordination.java:231)












































Like
(1)







Dislike
(0)



Favorite(1)
Share

Report















        
        

                        
                

                     Previous: The Service Becomes Abnormal After HBase Table Data Is Manually Deleted
                

            
                                    
                

                    Next: HBase SystemTable namespace Is Deleted Due to Misoperation
                

                
            
                    

        











Featured Answers










    

        

        

            







Best answer 














(2)



(0)
















            

            






                
tanglei
                

                    
                	

                    





                    
Created Jul 25, 2019 06:50:59
                    


                    




                	

                	                    


                    

                
                








 


        
                            
 
                                    

                                                                                                                        
Hello!

Troubleshooting guideline

Stop the HBase service, restore the table description information from another normal cluster and restart the HBase service.

Operation procedure

1. Stop the HBase service.

http://10.88.194.32:7088/idp-edit-service/editor/image/11108756775/A-1_1_en-us_image_0117892688.jpg

2. Run the following command to go to the installation directory on the client, for example, /opt/client:

cd /opt/client

3. Run the source bigdata_env command to configure environment variables.

4. If the cluster uses the security version, you need to perform security authentication.

kinit hbase

5. Enter the correct password as prompted.

[root@10-10-10-11 client]# source bigdata_env  #Configure environment variables.
[root@10-10-10-11 client]# kinit hbase         #Configure kinit authentication.

Password for hbase@HADOOP.COM:

6. Upload the .tableinfo.0000000001 table description information to /hbase/data/hbase/namespace/.tabledesc/.

[root@10-10-10-11 client] # hdfs dfs -put .tableinfo.0000000001 /hbase/data/hbase/namespace/.tabledesc/

7. Change the permission to hbase supergroup.

[root@10-10-10-11 client] # hdfs dfs -chown hbase:supergroup /hbase/data/hbase/namespace/.tableinfo.0000000001

8. Restart the HBase service to clear the table information cached in the HBase memory.

Verification

1. Perform the hbase hbck check:

hbase hbck hbase:namespace

[root@10-10-10-11 client]# hbase hbck hbase:namespace          #Check the status … …
  
Summary

Table hbase:meta is okay. 
    Number of regions: 1 
    Deployed on:  hadoop01,21302,1520492976636
Table hbase:namespace is okay. 
    Number of regions: 1 
    Deployed on:  hadoop02,21302,1520492979895
0 inconsistencies detected. 
Status: OK

Check that the HBase service is normal.

http://10.88.194.32:7088/idp-edit-service/editor/image/11108756775/A-1_1_en-us_image_0117892690.jpg
                            
                                

    

    

        

        

            
            View more
        

        

    

            
    
    
    
    











            


    
















    

  • x
    • 

  • convention：
    • 




























    
 


    
































                    












All Answers










    

        

        

            














(2)



(0)










2#







            

            






                
tanglei
                

                    
                	

                    

tanglei


                    
Created Jul 25, 2019 06:50:59
                    


                    




                	

                	                    


                    

                
                








 


        
                            
 
                                    

                                                                                                                        
Hello!

Troubleshooting guideline

Stop the HBase service, restore the table description information from another normal cluster and restart the HBase service.

Operation procedure

1. Stop the HBase service.

http://10.88.194.32:7088/idp-edit-service/editor/image/11108756775/A-1_1_en-us_image_0117892688.jpg

2. Run the following command to go to the installation directory on the client, for example, /opt/client:

cd /opt/client

3. Run the source bigdata_env command to configure environment variables.

4. If the cluster uses the security version, you need to perform security authentication.

kinit hbase

5. Enter the correct password as prompted.

[root@10-10-10-11 client]# source bigdata_env  #Configure environment variables.
[root@10-10-10-11 client]# kinit hbase         #Configure kinit authentication.

Password for hbase@HADOOP.COM:

6. Upload the .tableinfo.0000000001 table description information to /hbase/data/hbase/namespace/.tabledesc/.

[root@10-10-10-11 client] # hdfs dfs -put .tableinfo.0000000001 /hbase/data/hbase/namespace/.tabledesc/

7. Change the permission to hbase supergroup.

[root@10-10-10-11 client] # hdfs dfs -chown hbase:supergroup /hbase/data/hbase/namespace/.tableinfo.0000000001

8. Restart the HBase service to clear the table information cached in the HBase memory.

Verification

1. Perform the hbase hbck check:

hbase hbck hbase:namespace

[root@10-10-10-11 client]# hbase hbck hbase:namespace          #Check the status … …
  
Summary

Table hbase:meta is okay. 
    Number of regions: 1 
    Deployed on:  hadoop01,21302,1520492976636
Table hbase:namespace is okay. 
    Number of regions: 1 
    Deployed on:  hadoop02,21302,1520492979895
0 inconsistencies detected. 
Status: OK

Check that the HBase service is normal.

http://10.88.194.32:7088/idp-edit-service/editor/image/11108756775/A-1_1_en-us_image_0117892690.jpg
                            
                                

    

    

        

        

            
            View more
        

        

    

            
    
    
    
    











            


    
















    

  • x
    • 

  • convention：
    • 




























    
 


    
































                    


































Back to list











Comment

































Advanced



B
Color
Image
Link
Quote
Code
Smilies






You need to log in to comment to the post Login | Register














Comment







































    

         Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:

    
    
        
  • Politically sensitive content
    • 
        
  • Content concerning pornography, gambling, and drug abuse
    • 
        
  • Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy
    • 
    

Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " User Agreement."
    












My Followers







































     







Login and enjoy all the member benefits

Login















Huawei Website
Support
About Huawei
Privacy
User Agreement
Terms of Use
Cookies
Cookie Settings




Huawei Enterprise
Huawei Carrier Forum
Training & Certification





Contact Us:  e_online@huawei.com   Copyright © 2022 Huawei Technologies Co., Ltd. All rights reserved.









APP



    

    • 

    • 

    • 

    • 






















    

        Block
        
    

    

        

            Are you sure to block this user?
Users on your blacklist cannot comment on your post,cannot mention you, cannot send you private messages. 
        

    

    

        

            
            
        

    








    

        Reminder
        
    

    

        

            Please bind your phone number to obtain invitation bonus.