Problem Symptom
The upgrade fails due to a faulty node.
Problem Diagnosis
The upgrade is paused, and the faulty node is displayed on DeviceManager.
Log in to the active FSM node and run the ping Management plane IP address of the faulty node command to test the network connectivity. If the network is abnormal, perform the following operations.
Log in to the faulty node, view the tail -n100 /opt/fusionstorage/deploymanager/clouda/data/operation_result.txt file, view the fail record, and locate the fault cause based on the component error message. If the fault cannot be rectified, perform the following operations.
Causes
The faulty node hardware becomes faulty or the network is abnormal.
Solution
notice: This section describes how to temporarily skip the upgrade of the faulty node. After the upgrade is successful, you need to rectify the faulty. For example, remove the faulty node and add a new node.
Troubleshooting
Log in to DeviceManager and choose Cluster > Control Cluster. In the Selected Nodes area, check whether the faulty node is in the control cluster. If the faulty node is a management node or a control node, contact R&D engineers to determine whether the node fault can be ignored based on the current upgrade phase.
Log in to the active FSM node and switch to user root.
Run the cd /opt/fusionstorage/deploymanager command to go to the directory.
Run the python manul_ignore_hosts.py add_ignore command and enter the management floating IP address, user name and password for logging in to DeviceManager, and manegement plane IP address of the faulty node as prompted.
If the target version is earlier than 8.1.0, run the scp command to copy /opt/fusionstorage/deploymanager/.ignore_hosts to the /opt/fusionstorage/deploymanager directory on the standby FSM node. Log in to the standby FSM node, run the cd /opt/fusionstorage/deploymanager command to go to the directory, and run the chown fdadmin:ops .ignore_hosts and chmod 750 .ignore_hosts commands.
If the target version is 8.1.0 or later, run the scp command to copy /opt/fusionstorage/deploymanager/ignore_hosts to the /opt/fusionstorage/deploymanager directory on the standby FSM node. Log in to the standby FSM node, run the cd /opt/fusionstorage/deploymanager command to go to the directory, and run the chown fdadmin:ops ignore_hosts and chmod 750 ignore_hosts commands.
On the Upgrade page of DeviceManager, click Retry. After the upgrade is successful, repeat 2 and 3 and run the python manul_ignore_hosts.py del_ignore command to delete the failure records.
Enter the management floating IP address, user name and password for logging in to DeviceManager, and the management plane IP address of the node as prompted.
Handle the faulty node, for example, remove the faulty node from the storage pool and then expand the storage pool capacity.
Check After Recovery
Check whether services are running properly.
Suggestion and Summary
N/A
Applicable Versions
FusionStorage 8.0.1; FusionStorage 8.0.1.2; OceanStor 100D 8.0.2; OceanStor 100D 8.0.3
