Problem Information
Item | Description |
Storage type | Distributed storage |
Product version | FusionStorage 8.0.0 and later |
Problem type | Value-added feature |
Keyword | HyperMetro interruption; Remote device disconnection |
Problem Symptom
An alarm is generated indicating that a HyperMetro pair or consistency group is disconnected. The HyperMetro pair or consistency group is in FAULTY status and the running status is To be synchronized.
Problem Diagnosis
On DeviceManager, choose Monitor > Alarms and Events > Alarms and check whether an alarm is generated indicating that the remote device is disconnected.
If yes, go to 2.
If no, the fault is caused by other issues. This document is not applicable.
Locate the active replication node, run the diagnose_usr --set-cli command on any replication node, and run the ls command to check lsid of the dms process. In the following example, the lsid value is 125.
Run the rsf showcls command to check the IP address of the active node.
Log in to the active replication node as user root and run the following command to check whether any information is displayed:
zgrep -a ngcDevHeartbeatTimer /var/log/dr/rep/{bak,run}/* | grep "loss heartbeat"If yes, the replication link heartbeat is lost due to a network fault. Go to 1.
If no, go to 3.
Run the following command on all replication nodes as user root to check whether any information is displayed:
zgrep -a "lost cm connect" /var/log/dr/rep/{bak,run}/* | grep "kill process"If yes, the replication process exits due to heartbeat loss between the cm client and cm server. Go to 1.
If no, go to 4.
Run the following command on each replication node to check whether the drop rules exist:
iptables -L | grep 12100
If yes (as shown in the red box in the preceding figure), the replication link is disconnected due to the firewall. Go to 2.
If no, the fault is caused by other issues. This document is not applicable.
Causes
The replication link between clusters is disconnected, causing the disconnection of HyperMetro pairs.
Solution
If scenarios in steps 2 and 3 in Problem Diagnosis occur, rectify the network faults. After rectification, wait for 2 or more minutes and then check whether the alarm indicating remote device interruption is cleared.
If yes, no further action is required.
If no, go to 3.
If the scenario in step 4 in Problem Diagnosis occurs, run a command to clear the drop rules of the firewall. For example:
iptables -D OUTPUT -p tcp --dport 12100 -j DROP
After clearing on all nodes, wait for 2 or more minutes and then check whether the alarm indicating remote device interruption is cleared.
If yes, no further action is required.
If no, go to 3.
Contact technical support engineers.
Check After Recovery
On DeviceManager, check whether the alarm indicating remote device interruption is cleared and whether the status of the HyperMetro pair or consistency group is Synchronizing or Normal.
Suggestion and Summary
N/A
Applicable Versions
All
