Problem Symptom
During the upgrade process, a component of a node fails to be upgraded, and the upgrade still fails after retry.
Problem Diagnosis
If the upgrade fails, click Retry. The upgrade task still fails.
Check the upgrade status on DeviceManager, and determine whether the upgrade failure occurs only on a component of a node.
After confirming the faulty node, log in to the node to view the upgrade logs of the faulty component (the logs of each component are all stored in the /var/log directory and can be queried by component name), and check whether the component can be restored within a short period of time.
Rectify the fault based on the following operations if the component cannot be rectified within a short period of time.
Causes
A component of a node fails to be upgraded. As a result, the upgrade of the system fails.
Solution
Note: This section is only about how to temporarily skip the upgrade of the faulty node. After the upgrade task is complete, you need to log in to the faulty node and manually upgrade the node.
Troubleshooting
Locate the fault cause and check whether the upgrade failure is caused by an abnormal component on a node.
Check whether the faulty component can be rectified immediately, and thereby determine whether the upgrade can continue after retry.
If the faulty component cannot be rectified immediately, record the timeline during which the node is faulty, including the components that have been upgraded and the components that fail to be upgraded.
Manually run the touch /home/.preinstall command on the faulty node to skip the upgrade of the faulty node and continue the upgrade task.
Re-perform the upgrade task and ensure all nodes are successfully upgraded.
Fault Rectification
Log in to the faulty node and switch to user root. If you cannot log in to the node, contact technical support engineers.
Go to directory cd /opt/fusionstorage/deploymanager/clouda/bin, run script vim manul_upgrade_for_node.py, use # to comment out the 360th line exit(0), and save the modification.
Run the following command on the faulty node:
cd /opt/fusionstorage/deploymanager/clouda/bin; python manul_upgrade_for_node.py fix
Enter the management floating IP address, user name and password for logging in to DeviceManager, and IP address of the faulty node as prompted.
Click Retry.
Do not click Confirm after the upgrade.
Run the following command on the faulty node:
cd /opt/fusionstorage/deploymanager/clouda/bin
Command python manul_upgrade_for_node.py fix is used to manually upgrade the components that are not upgraded. Then enter the management floating IP address, user name and password for logging in to DeviceManager, and IP address of the faulty node as the previous step.
Click Confirm.
Check After Recovery
Re-perform the upgrade task, and all nodes are successfully upgraded.
Suggestion and Summary
N/A
