Symptoms
After running complicated services (such as slow disk injection, SAN+NAS, and link fault injection) for a while, there is slight possibility that the following errors occur: Files to be deleted do not exist and files to be created exist.
Alarm Information
No alarms
Possible Causes
Reconnection of a TCP link between the storage device and a client causes the processed packets that have not been sent on the link to be discarded. As a result, the new TCP link tries to send the packets again. If the packets are about Create and Remove operations, the storage device reports errors because these operations have been performed. If the packets are about other operations, the storage device does not report errors because it simply re-performs the operations.
Identification Method
Monitor changes of the client's TCP ports, collect the client's RPC logs, or capture packets. If any of the following is discovered:
A TCP link is disconnected and reconnected, causing a TCP port change.
RPC: xs_tcp_send_request(40432) = -11 exists in RPC logs.
In packets, it is discovered that the client sends FIN packets to the storage device.
Then it can be confirmed that the issue is caused by TCP link disconnection on the client.
Troubleshooting Procedure
There are many possible causes for TCP link disconnection. You can advise to customer to:
Check the service model whether the client carries excessive services. This causes packet accumulation and link disconnection.
Check the network whether packets accumulate and the link is disconnected due to the network.
Check the client running status whether another issue makes the client run slowly and packets accumulate, which in turn causes link disconnection.
Check After Recovery
Services are running correctly, without any error displayed.
Application Scope
OceanStor V3 series V300R002
OceanStor V3 series V300R003
