Hi, dear!
Good day to you!
In this post, I will explain how to deal with the CMU offline issue.
Problem Description
CMUoffline alarms.
Problem Analysis
According to the alarm details，4 VCN repeatedly generated these alarms.
CMU is the module for managing clusters, checked the cluster status in on OMU portal and found that the cluster had expired, the cluster member status was unknown.
Analyzed the CMU logs and found there was zookeeper disconnection error.
Root Cause
Zookeeper disconnection error occurred, leading to cluster exception.
Solution Description
1. Checked the time of the 4 VCN had been synced.
2. Logged into the 4 VCNservers and run the command df -h to check the space usage: no partition was full.
3. Run the command: /home/ivstool/bin/service.shrestart cmu to restart the CMU process for all the 4 VCN.
4. Run the command: ps -ef|grepzookeeper to check the pid number of the zookeeper process on the 4 VCN.
5. Run the command: kill the pid number checked in step4 to kill and restart the zookeeper process on the 4VCN.
6. After the above operations, the cluster status showing on OMU portal changed to normal and the alarms on eSight were cleared.
Thanks for reading!
