Failed to complete the instances start operation

On the FusionInsight Manager HDFS service page, the two NameNodes are always in the Standby state and cannot provide services.

Possible Causes

• Active NameNode selection fails because the old data that controls the active/standby status of NameNodes remains in ZooKeeper.

• The session ID of the ZooKeeper client ZKFC is inconsistent with that of the ZooKeeper server, resulting in connection deadlock.

• JournalNode is abnormal and an active/standby switchover occurs on the NameNodes. When editlog is being restored on the NameNodes, an editlog synchronization failure occurs.

Troubleshooting Method

• In the ZKFC logs, if information similar to java.lang.IllegalArgumentException: Unable to determine service address for namenode '1781' exists, the old data remains in ZooKeeper.

• In the logs of the two ZKFCs, if the "Active > Standby" and "Standby > Active > Standby" NameNode status switchovers exist, the ZooKeeper client ZKFC and the ZooKeeper server are in deadlock connection.

• On the NameNodes, write editlog and restore editlog failures exist.

1. In the logs of the NameNode that is supposed to be in the Active state, if the following error information exists:

   Got too many exceptions to achieve quorum size 2/3.

   a write editlog failure occurs, the NameNode restarts, and the "Active > Standby" NameNode status switchover exists.

2. In the logs of the NameNode that is supposed to be in the Standby state, if the following error information exists:

   recoverUnfinalizedSegments failed for required journal

   a restore editlog failure occurs, the NameNode restarts, an Active status preemption failure occurs, and the NameNode is in the Standby state.

Procedure

• The old data remains in ZooKeeper.

1. Log in to the node where the ZooKeeper client is located as user root.

    NOTE:

   Install the ZooKeeper client. For details, see section Installing a Client in the Software Installation.

2. Run the following command to export the environment variables:

   cd ZooKeeper client installation directory

   source bigdata_env

3. In security mode, run the following command to log in to Kerberos as user hdfs (skip this step in normal mode).

   kinit hdfs

    NOTE:

   Contact the cluster administrator to obtain the password of user hdfs. The default password is Hdfs@123.

4. Use the ZooKeeper client to access the CLI.

   zkCli.sh -server service IP address of any functioning ZooKeeper node:24002

5. Run the following command to delete the old data:

   deleteall /hadoop-ha/hacluster/ActiveBreadCrumb

6. Check whether the active/standby status of NameNodes is recovered.

The ZooKeeper client and server are in deadlock connection.

1. Log in to FusionInsight Manager, choose Services > HDFS > NameNode, and select the NameNode in the Standby > Active > Standby state.

2. Choose More Actions > Restart instance to restart the NameNode.

On the NameNodes, write editlog and restore editlog failures exist.

Perform the following operations on the two NameNodes to restore editlog.

• In security mode, run the following commands:

  cd ${BIGDATA_HOME}/FusionInsight/1_6_NameNode

  cp ${BIGDATA_HOME}/FusionInsight/install/FusionInsight-Hadoop-2.7.2/hadoop/sbin/exportENV_VARS.sh ./

• In normal mode, run the following commands:

  cd ${BIGDATA_HOME}/FusionInsight/1_6_NameNode

  cp ${BIGDATA_HOME}/FusionInsight/install/FusionInsight-Hadoop-2.7.2/hadoop/sbin/exportENV_VARS.sh ./

1. On FusionInsight Manager, stop HDFS.

2. Log in the new NameNode node as user root, run the su - omm command to switch to user omm.

3. Run the following commands to export the environment variables:

   EXECUTABLE_HOME="${CONTROLLER_HOME}/kerberos_user_specific_binay/kerberos"

   LD_LIBRARY_PATH=${EXECUTABLE_HOME}/lib:$LD_LIBRARY_PATH

   PATH=${EXECUTABLE_HOME}/bin:$PATH

4. (Optional) In security mode, run the kinit command to obtain a security certificate.

   kinit hdfs

5. Go to the ${BIGDATA_HOME}/FusionInsight/1_6_NameNode directory, and copy the environment variable script to the directory. The number 1_6 in the directory changes as the environment changes.

6. Run the following command for the environment variables to take effect.

   source exportENV_VARS.sh

7. Run the following command to restore editlog.

   cd $HADOOP_HOME/bin

   ./hdfs namenode -recover

8. Enter y as prompted.

9. On FusionInsight Manager, start HDFS.