Invalid argument faults occured in OceanStor 9000 when run the cmd "ls" about a directory mounted by NFSv3

1、  According to the packets captured at the client side, the Linux command “ls” doesn’t receive any responses and tries to retransmit the request.

2、  With the further analysis from the logs, we find that the unlock message OPEN_LOCK produced by the Linux command ‘ls’ is not replied by the lock server that will decide which node to do the unlock operation, so that the client receives no response. When open a file or directory, OPEN_LOCK will be used.

3、  In accordance with the logs in the protocol module, when the number of re-transmissions received by the OceanStor 9000 has reached the limit of timeout, the OceanStor 9000 will reply the NFS3ERR_INVAL back to the client. It means invalid argument or unsupported argument for an operation, which is caused by the timeout.

4、  With the analysis of the logs of the lock module, we find that there is a RPC link problem from the node4 to node4, which leads to no response to unlock message. When one node starts, it will create RPC links from this node to all nodes to transfer the unlock command. All nodes are lock server. Concretely, when a client runs the command “ls” the S002 directory by IPs of node4, node4 will get lock from a lock server in order to access the directory. But the lock server is just node4.

In summary, the issue is caused by the problem of the RPC link from node4 to node4. When the Linux command ‘ls’ tries to access the certain directory in the NFSv3 share, the system decides the RPC link from node4 to node4 will be used which is faulty. Besides, the reason why NFSv4 goes well is because NFSv4 process doesn’t use the lock module.

Root Cause

The issue was caused by problematic RPC link that is used to get lock. So that when "ls" a directory, it can't get the lock from a lock server. It’s a very low probability bug to create a problematic RPC link.

Solution

[Workaround]
Mount NFS shares by NFS4 or by frontend IPs of other nodes.

[Future solution]
The issue is resolved by the latest version V100R001C30SPC200. You can consider an upgrade after the version of software becomes the recommended version on Huawei website.