Problem Information
Item | Description |
Storage type | Distributed storage |
Product version | FusionStorage OBS 7.0; FusionStorage 8.0.0; FusionStorage 8.0.1 |
Problem type | Capacity expansion |
Keyword | Microservices; suspension; time hopping |
Problem Symptom
No progress is displayed on the microservice capacity expansion page of DeployManager for more than 30 minutes.
Problem Diagnosis
Log in to the active node of DeployManager as user dfvmanager using SSH, and then switch to user root.
Continuously observe the scheduled task logs of celery to check whether Sending due task query_doing_task_result is updated. Example:
tailf /var/log/servicetool/CloudAutoDeploy/sdk/celerybeat.log
If Sending due task query_doing_task_result is not updated for five minutes, the capacity expansion task is suspended. Otherwise, no further action is required.
Causes
Time hopping occurs on the node.
Solution
Log in to the active node of DeployManager as user dfvmanager using SSH, and then switch to user root.
Delete the celerybeat-schedule file.
rm /var/log/servicetool/CloudAutoDeploy/sdk/celerybeat-schedule
Restart the celery process.
sh /opt/admin/CloudAutoDeploy/deploy/sdk/deployscripts/clyservice stopsh /opt/admin/CloudAutoDeploy/deploy/sdk/deployscripts/clyservice start
Check After Recovery
Log in to DeployManager again.
Go to the capacity expansion page and check whether the expansion task is recovered.
Suggestion and Summary
N/A
Applicable Versions
All
