Got it

Emergency Recovery of Hive and Spark Service Status Faults Due to Account Lockout

Latest reply: Nov 21, 2021 14:21:47 365 4 2 0 0

Hello, everyone!

I'm going to introduce you to a method of Emergency Recovery of Hive. It helps beginners learn about FI faster.

Applicable Versions

6.5.x

Symptom

The Hive and Spark services reported by the FusionInsight cluster are unavailable. The metastore instances are faulty.

The error message 'PSQLException: FATAL: The account has been locked.' is displayed in the metastore log (/var/log/Bigdata/hive/metastore/metastore.log) of any node.

2018-05-03 14:53:08,746 | ERROR | BoneCP-pool-watch-thread | Error in trying to obtain a connection. Retrying in 1000ms | com.jolbox.bonecp.PoolWatchThread.fillConnections(PoolWatchThread.java:118) 
org.postgresql.util.PSQLException: FATAL: The account has been locked. 
        at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:366) 
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:180) 
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:64) 
        at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:124) 
        at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:28) 
        at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:20) 
        at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:30) 
        at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:22) 
        at org.postgresql.Driver.makeConnection(Driver.java:392) 
        at org.postgresql.Driver.connect(Driver.java:266) 
        at java.sql.DriverManager.getConnection(DriverManager.java:664) 
        at java.sql.DriverManager.getConnection(DriverManager.java:208) 
        at com.jolbox.bonecp.BoneCP.obtainRawInternalConnection(BoneCP.java:361) 
        at com.jolbox.bonecp.BoneCP.obtainInternalConnection(BoneCP.java:269) 
        at com.jolbox.bonecp.ConnectionHandle.<init>(ConnectionHandle.java:242) 
        at com.jolbox.bonecp.PoolWatchThread.fillConnections(PoolWatchThread.java:115) 
        at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:82) 
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)

Impact and Severity

The Hive and Spark services in the FusionInsight HD cluster cannot provide services.

Expected Recovery Duration

20 minutes

Prerequisites

This operation has had a great impact. Before performing this operation, you must confirm with the R&D engineers for approval. Prepare the following items before the restoration.

Table1 Items to be prepared before the restoration

No.

Project

Operation

1

Cluster account information

Apply for the password of cluster user admin.

2

Node account information

Apply for the passwords of users omm and root of cluster nodes.

3

Secure Shell (SSH) remote login tool

Prepare tools, such as PuTTY or SecureCRT.

4

Cluster account information

Administrator of GaussDB in the DBService component


Fault Handling

Log in to the Dbservice background and manually unlock the Hive account.


Operation Guide

The C60U10 version is used as an example. The paths of the .dbservice_profile files of other versions are different.

C70SPC20* or later: /opt/huawei/Bigdata/FusionInsight_BASE_*/*_*_DBServer/install/.dbservice_profile

C60U20*/C60U10*/C60SPC20*: /opt/huawei/Bigdata/FusionInsight/dbservice/.dbservice_profile

C50SPC20*:/opt/huawei/Bigdata/dbservice/.dbservice_profile


1. Log in to the FusionInsingh Manager as user admin and obtain the management IP address of the node where the active DBServer resides.

FusionInsingh Manager

2. Log in to the node where the active DBServer is located as user omm and run the following command:

For example, in the C60 environment:

omm@hadoop02:~> source /opt/huawei/Bigdata/FusionInsight/dbservice/.dbservice_profile

3. Run the gsql -p 20051 -U omm -d postgres -W ommHuawei@123 -c 'alter role HIVE account unlock'; command as user omm.

In the preceding command, omm is the administrator of GaussDB in the DBService component, and the password of the administrator account:

C60: ommHuawei@123

C70 and C80: dbserverAdmin@123

For example:

[omm@189-X-X-106 ~]\>source /opt/huawei/Bigdata/FusionInsight/dbservice/.dbservice_profile 
[omm@189-X-X-106 ~]\>gsql -p 20051 -U omm -d postgres -W ommHuawei@123 -c 'alter role HIVE account unlock';
ALTER ROLE
[omm@189-X-X-106 ~]\>

That's all, thanks!

  • x
  • convention:

Helen.F
Created Aug 24, 2021 02:59:15

Thanks for your sharing!
View more
  • x
  • convention:

Unicef
MVE Created Aug 24, 2021 03:22:33

Cool thanks
View more
  • x
  • convention:

olive.zhao
Admin Created Nov 17, 2021 02:56:53

Thanks for your sharing!
View more
  • x
  • convention:

user_4358465
Created Nov 21, 2021 14:21:47

Informative post! Thank you!
View more
  • x
  • convention:

Comment

You need to log in to comment to the post Login | Register
Comment

Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:
  • Politically sensitive content
  • Content concerning pornography, gambling, and drug abuse
  • Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy
Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " User Agreement."

My Followers

Login and enjoy all the member benefits

Login

Block
Are you sure to block this user?
Users on your blacklist cannot comment on your post,cannot mention you, cannot send you private messages.
Reminder
Please bind your phone number to obtain invitation bonus.
Information Protection Guide
Thanks for using Huawei Enterprise Support Community! We will help you learn how we collect, use, store and share your personal information and the rights you have in accordance with Privacy Policy and User Agreement.