[Problem Description]
Third-party NMS (zabbix, snmpwalk, snmpget, eservice, mib browser, etc.) Access to the storage array through SNMP does not respond.
[Fault Symptom]
Zabbix and eService monitor storage devices using SNMP. Storage information cannot be queried using the snmpwalk command. Tip: Timeout: No Response from XXX.
[Cause]
1. Direct cause: When SNMP uses a third-party library, threads in the third-party library are locked incorrectly. As a result, thread deadlock occurs.
2. Root cause: Multiple threads are used to process SNMP requests in the third-party SNMP library. During locking, the lock is repeatedly released. As a result, the MIB tree update is not locked, and the memory fatal lock is executed.
[Location Method]
1. Capture packets on the storage device to check whether the storage device responds.
Log in to the bootrom system and run the tcpdump.sh -i any port 161 command. If the SNMP port has been changed, change the port 161 to the specified port.
The command output shows that port 161 of the disk array does not respond.
2. Export the log package and check whether SNMP logs are recorded. If the SNMP is not suspended, consecutive logs are generated. The log directory is as follows:
The preceding figure shows the log screenshot. If the last log is long later than the current log, the process is suspended.
[Solution]
The converged storage array is upgraded to V5R7C30SPH109/V3R6C50SPH109.
Upgrade the Dorado array version to Dorado V3R2C10SPC100SPH110.
[Workaround]
You can run the change snmp status status=stop command.
Run the change snmp status status=start command to restart the SNMP service. This problem may occur again with the customer's query frequency.
[Recovery method check]
You can run the mib browser, zabbix, and snmpwalk commands to access the disk array. If the access is successful, the disk array is restored.
[Suggestion and Summary]
NA
[Applicability]
OceanStor V5R7C00/V3R6C10
OceanStor V5R7C10/V3R6C20
OceanStor V5R7C20/V3R6C30
OceanStor V5R7C30/V3R6C50
Dorado V3 V3R1C00----V3R2C20
