Emergency Handling for a VBS Process in the D State

Problem Information

Problem Symptom

In the Linux OS, SCSI volumes are mounted to the node for upper-layer applications, and file systems EXT3/EXT4/XFS are created. During the operation, services are interrupted, and the VBS process enters the D state.

Problem Diagnosis

1. When services are interrupted, run the top command to check whether the VBS process is in the D state.

   
   

2. Check the system logs (/var/log/message) before and after the problem occurs. It is found that the VBS call stack is printed and the rwsem_down_read_failed/rwsem_write_read_failed function fails.

   
   

3. Verify the fault based on the preceding two steps.

Causes

The VBS process accesses the memory page and enters the kernel. After the kernel occupies the memory management lock, the memory is insufficient, triggering cache reclamation. The cache reclamation process causes dirty data to be flushed to disks. The VBS node fails to enter the kernel and stays in the D state.

Solution

1. Use PuTTY to log in to the node where the abnormal VBS process resides and run the top command to check the VBS process status. The VBS process stays in the D state for more than 1 minute.

   
   

   If yes, go to 2.

   If no, no further action is required.

   
   

   
   

2. With the customer's permission, run the echo b >/proc/sysrq-trigger command to restart the faulty node to restore services. During the restart, data cached in the file system will be partially lost. The amount of lost data depends on the file system type of the faulty node. If the customer does not agree to restore services by restarting the faulty node, contact technical support engineers.

Check After Recovery

After the faulty compute node restarts, log in to the node and check whether the dsware_vbs process is in the S or R state. If yes, the VBS process is rectified. Then, ask the customer to confirm the rectification.

Suggestion and Summary

N/A

Applicable Versions

All