A ZooKeeper Node Fails to be Migrated Due to a Fault Occurred on the Node Before the Migration

Problem Information

Problem Symptom

A ZooKeeper node is faulty. As a result, it fails to be migrated.

Problem Diagnosis

The parent migration task fails. After the subtask of updating the database or writing ZooKeeper configurations to the MDC node is complete, no subsequent subtask exists.

Causes

During the migration, the network or node is abnormal, and the communication with the source ZooKeeper node fails. As a result, subsequent subtasks time out and fail.

Solution

If the source ZooKeeper node is permanently faulty and cannot be restored, this section is not applicable.

If the source ZooKeeper node can be restored, perform the following steps after the node is restored:

Use PuTTY to log in to the active FSM node through the management plane floating IP address and switch to user root.

Run the ismcli -u admin command and enter user name admin and its password to log in to the CLI.

1. Run the following restoration commands:

   change cluster_profile update cluster_type=manager process=zk

   change cluster_profile update cluster_type=manager process=mdc

   change cluster_profile update cluster_type=eds process=cm

   change cluster_profile update cluster_type=rep process=cm

2. Log in to the active FSM node as user dsware, switch to the /opt/dsware/client/bin directory, and run the following dswareTool command to clear configuration of the source ZooKeeper node: sh /opt/dsware/client/bin/dswareTool.sh --op clearNode -type zk -ip ${zk error node ip}

   

Check After Recovery

1. Use PuTTY to log in to the faulty node and run the ps -ef | grep zk command to check whether the source ZooKeeper process has been cleared.

2. Use PuTTY to log in to the new ZooKeeper node and run the ps -ef | grep zk command to check whether the new ZooKeeper process has been started.

3. Use PuTTY to log in to the original MDC node and run the cat /opt/fusionstorage/persistence_layer/mdc/conf/mdc_conf.cfg |grep mdc_zk_address command to check whether the IP address of the ZooKeeper node in the MDC configuration file is updated to that of the new ZooKeeper node.

4. Use PuTTY to log in to the original ZooKeeper node and run the cat /opt/fusionstorage/persistence_layer/agent/zk/conf/zoo.cfg |grep server command to check whether the IP address of the ZooKeeper node in the ZooKeeper configuration file is updated to that of the new ZooKeeper node.

Suggestion and Summary

N/A

Applicable Versions

All