Hi team!
Here’s a case that Toolkit inspection fails due to 100% CPU usage of the Master Controller.
Symptom
1.An alarm is displayed on the GUI of DeviceManger indicating that the CPU usage of controller A reaches 100%.
2.During a Toolkit inspection, the CPU usage inspection item fails.
Fault Description
The Toolkit inspection fails due to 100% CPU usage of the controller A.
Cause
1.The residual CLI processes cause the CPU usage of controller A to be 100%.
There are 32 ismcli processes, and each process occupies 5.7% of the CPU.
2.The residual CLI processes exist because the password expires and the system displays a page of modifying the password.
However, the request is not processed.
Each time when you close the page, one CLI residual process is generated.
The system allows you to close the page for 32 times in maximum.
Therefore, there a 32 residual processes in maximum.
Fault Diagnosis
In the /other/cur_task_info master controller log file, enter the cli keyword to search for information.
If there are a large number of /ISM/cli/ismc processes, the fault is diagnosed.
For details, see the following screenshot.
Solutions
1.Workarounds:
（1）Log in to the command-line interface (CLI) using PuTTY or SSH, and run the following command to go to the developer mode.
The debug password is debug@storage.
change user_mode current_mode user_mode=developer
（2）Go to the minisystem mode.
（3）Run the ps –C ismcli command to query PID of a CLI residual process.
（4）Run the kill -9 PID command to kill related processes.
Note: In the result queried in the preceding step, find all the PID processes whose values in the TTY column are a quotation mark (?) and kill the processes in sequence.
Exit the CLI after all the PID processes whose values in the TTY column are a quotation mark (?) are killed.
2.Preventive measures
(1)Run the show upgrade package command to check the current versions of the storage system and patch.
The version of OceanStor 18000 series is V100R001C00 SPC300.
The version of OceanStor T series is V200R002C00 SPC400.
(2)After the versions are verified, install patches of corresponding versions.
Install the V1R1C00SPH303 version patch for OceanStor 18000 V100R001 series.
Install the V2R2C00SPH403 version patch for OceanStor T V200R002 series.
Check After Recovery
1.The alarm indicating that the CPU usage of controller A reaches 100% is resumed.
2.The CPU usage inspection item passes in the Toolkit inspection.
