CloudA Fails to Be Automatically Upgraded Due to a Node Fault

Problem Symptom

During the CloudA automatic upgrade, the node is restarted or faulty, and upgrade fails. The following information is displayed:

[ERROR] upgrade clouda failed!

clouda upgrade result is :1

CloudA automatic upgrade log path: /var/log/deploy/scripts/package.log

Problem Diagnosis

1. Log in to any FSM node and switch to user root.

2. Export the node information:

   For FusionStorage 8.0.1 and OceanStor 100D 8.0.2, run the following commands:

   touch /tmp/sql_result.txt

   exec_sql="/opt/fusionstorage/deploymanager/gaussdb/app/bin/gsql -p 7018 -d cmdb -W Huawei12#$ -c \\\"select * from HOST_INFO;\\\""

   eval "su - dmdbadmin -c \"${exec_sql}\"" >> /tmp/sql_result.txt

   

   For OceanStor 100D 8.0.3 and OceanStor Pacific series 8.1.0, and OceanStor Atlantic series 8.1.0, run the following commands to export the node information:

   touch /tmp/sql_result.txt

   exec_sql="/opt/fusionstorage/deploymanager/gaussdb/app/bin/gsql -U dmdbadmin -d cmdb -W Huawei12#$ -c \\\"select * from HOST_INFO;\\\""

   eval "su - dmdbadmin -s /bin/bash -c \"${exec_sql}\"" >> /tmp/sql_result.txt

   

3. Run the following command to view the exported node information and obtain the information about the node where CloudA fails to be automatically upgraded (the value of AGENT_STATUS is 3):

   vim /tmp/sql_result.txt

4. Log in to the faulty node and run the following command to check the CloudA upgrade result. In the end of the command output, "CloudA upgrade failed!" is not displayed:

   vim /var/log/cloudagent/clouda.script.log (applicable to FusionStorage 8.0.1 and OceanStor 100D 8.0.2)

   vim /var/log/deploy/clouda/script/clouda.deploy.script.log (applicable to OceanStor 100D 8.0.3 and OceanStor Pacific series 8.1.0, and OceanStor Atlantic series 8.1.0)

Causes

CloudA fails to be automatically upgraded due to a node restart or node failure.

Solution

1. Rectify the faulty node.

2. Log in to the node where CloudA fails to be upgraded and run the following commands to manually upgrade CloudA:

   cd /opt/fusionstorage/repository/deploymanager_pkg/micro/clouda_1.0.0/action/

   sh upgrade.sh

3. Run the following command to check the upgrade result. If "CloudA upgrade successfully!" is displayed, the upgrade is successful:

   vim /var/log/cloudagent/clouda.script.log (applicable to FusionStorage 8.0.1 and OceanStor 100D 8.0.2)

   vim /var/log/deploy/clouda/script/clouda.deploy.script.log (applicable to OceanStor 100D 8.0.3 and OceanStor Pacific series 8.1.0, and OceanStor Atlantic series 8.1.0)

Check After Recovery

Log in to any FSM node and run the following commands to check AGENT_STATUS of the faulty node. If the node recovers, its AGENT_STATUS is 1.

For FusionStorage 8.0.1 and OceanStor 100D 8.0.2, run the following commands to export the node status and view the status:

rm -rf  /tmp/sql_result.txt

touch /tmp/sql_result.txt

exec_sql="/opt/fusionstorage/deploymanager/gaussdb/app/bin/gsql -p 7018 -d cmdb -W Huawei12#$ -c \\\"select * from HOST_INFO;\\\""

eval "su - dmdbadmin -c \"${exec_sql}\"" >> /tmp/sql_result.txt

vim /tmp/sql_result.txt

For OceanStor 100D 8.0.3 and OceanStor Pacific series 8.1.0, and OceanStor Atlantic series 8.1.0, run the following commands to export the node status and view the status:

rm -rf  /tmp/sql_result.txt

touch /tmp/sql_result.txt

exec_sql="/opt/fusionstorage/deploymanager/gaussdb/app/bin/gsql -U dmdbadmin -d cmdb -W Huawei12#$ -c \\\"select * from HOST_INFO;\\\""

eval "su - dmdbadmin -s /bin/bash -c \"${exec_sql}\"" >> /tmp/sql_result.txt

Suggestion and Summary

N/A

Applicable Versions

All