HA Heartbeat Alarms of OAM-U Are Not Cleared

Hello, everyone!

HA heartbeat alarms of OAM-U are not cleared after network fault recovery or management IP address modification. Do you know the reason? Don't worry, the post will share with you!

Basic Information

Symptom

After the network fault is rectified for a period of time, an alarm indicating that the HA heartbeat is interrupted or the HA file fails to be synchronized is not cleared on the GUI.

Diagnosis

1. Log in to a management node as user root.

2. If the HA heartbeat interruption or synchronization failure alarm (error code 0xFC3670001 or 0xFC3670002) is not cleared, run the netstat -anp | grep 61592 and netstat -anp | grep 61593 commands to check whether the problem occurs. Run the preceding commands for multiple times and check whether SYN_RECV or SYC_SENT is displayed in the command output.

If yes, no further action is required.

If no, the fault is caused by other issues. This document is not applicable.

Cause

The client uses fixed ports for heartbeats and file synchronization between HA nodes. In scenarios where a switch is faulty, the management IP address is modified, or the network is faulty, the operating system cannot reclaim the ports after the fault recovers, resulting in HA nodes cannot establish new connections.

Solution

1. Log in to a management node as user root.

2. Run the ps -aux | grep oam-u/ha | grep ha.bin command to query the HA process PID.

3. Run the kill -9 PID command to kill the process.

4. Wait for a period of time and check whether the alarm is cleared. If the alarm persists, log in to the other management node and perform the preceding operations.

5. If the alarm persists, contact technical support engineers.

Check After Recovery

Check the alarm information. The HA heartbeat interruption alarm is clear.

This is my solution, how about yours? Go ahead and share it with us!