Failed to Start the EDS Process on an FSA Node After a Storage Pool Capacity Expansion Failure

Problem Information

Problem Symptom

During a capacity expansion, the new node or disk has been added to the storage pool, but the EDS process fails to be started on an FSA node. As a result, the capacity expansion fails.

Problem Diagnosis

1. Locate the MDC node and check whether the disk to be added exists in the storage pool.

   Run the following swareTool.sh command on the active FSM node:

   ./dswareTool.sh --op queryNodeProcessInfo

   

2. Log in to the MDC node and run the following dsware_insight command to check that the disk to be added is not in the storage pool, as shown in the following figure, the ESN of the disk is not displayed.

   dsware_insight 0 4 x.x.x.x 10530 8 168 1

   

   In the preceding command, 4 and x.x.x.x are listed as an example. Change the values as required. Run the following commands to obtain the actual values:

   cat /opt/fusionstorage/persistence_layer/mdc/conf/mdc_conf.cfg|grep mdc_id=

   cat /opt/fusionstorage/persistence_layer/mdc/conf/mdc_conf.cfg|grep mdc_ip_2=

3. Log in to the node to be added and check whether Agent has processed the OSD process, that is, check the OSD monitoring list.

   [root@ctup000102130 ~]# cat /opt/fusionstorage/agent/conf/agentMonitor

   [JBODMNG_PROCESS]

   jbodmng_is_monitor=True

   jbodmng_alarm_status=True

   process_name=JBODMNG

   [OSD_PROCESS]

   osd_1_3_is_monitor=True

   osd_1_3_para=10183204153xvde_66

   osd_1_3_out_reason=0

   osd_1_2_is_monitor=True

   osd_1_2_para=10183204153xvdf_65

   osd_1_2_out_reason=0

   osd_1_1_is_monitor=True

   osd_1_1_para=10183204153xvdg_64

   osd_1_1_out_reason=0

   osd_1_num=3

   osd_alarm_status=True

   process_name=OSD

Causes

After the capacity expansion fails, the exception is not handled for a long time. As a result, the MDC node rejects the corresponding disk.

Solution

• The following describes how to replace an SSD card or NVMe disk:

  Run the ./dswareTool.sh --op replaceSSD -id 1 -esn 10183204153xvdf:10183204153xvdf,10183204153xvde:10183204153xvde,10183204153xvdg:10183204153xvdg -nodeMgrIp x.x.x.x command.

  

  After the preceding command is successfully executed, run the following command to check whether the ESN of the disk to be added is in the storage pool:

  dsware_insight 0 4 x.x.x.x 10530 8 168 1

• Run the dswareTool command to expand the capacity. For details about related parameters, see the command description. The dsware_add.xml file only needs to contain the IP addresses.

  ./dswareTool.sh  --op scanMediaForPool ip x.x.x.x

  Generated file dsware.xml contains only the following IP address information and rename it as dsware_add.xml.

  ./dswareTool.sh --op addStorageNode -poolId 0 -mediaType ssd_card -cacheType none -f /opt/dsware/client/conf/dsware_add.xml

Check After Recovery

The capacity expansion is successful.

Suggestion and Summary

If the main storage type is SSD, SATA disk, or SAS disk, run the replaceDisk command.

Applicable Versions

All