Low Bandwidth of IB Port Links on a Dell R6525 Compute Node

Problem Information

Problem Symptom

Dell R6525 servers (AMD CPUs) function as compute nodes. Each server has one Mellanox CX-5 100 Gbit/s IB NIC (model: MCX555A-ECAT). Compute nodes and distributed storage nodes are interconnected through IB switches.

Use ib_send_bw in the perftest suite of the MLNX_OFED NIC driver to test the maximum bandwidth of physical links of the IB network ports on two compute nodes.

In a scenario where the I/O size is 4 KB or 1 MB and the number of QPs (RDMA connections) is 1 or 10, the maximum bandwidth of a physical link is far lower than 100 Gbit/s.

Problem Diagnosis

Check the MLNX_OFED driver version, DELL server mainboard BIOS parameter settings, and IB NIC firmware parameter settings.

Solution

1. X2APIC Mode is set to Disabled for the mainboard BIOS of the Dell R6525 server.

   

2. Restart the compute node.

3. Upgrade the IB NIC (Mellanox MCX555A-ECAT) firmware to the latest version (for example, 16.28.2006) provided by the Mellanox official website.

   #mst start

   #mst status

   #flint -d /dev/mst/mt4119_pciconf0 -i xxx.bin b

   

4. Restart the compute node.

5. Modify the configuration of the IB NIC firmware as follows:

   #mst start

   #mst status

   #mlxconfig -y -d /dev/mst/mt4119_pciconf0 q |grep PCI_WR

   

   #mlxconfig -y -d /dev/mst/mt4119_pciconf0 s PCI_WR_ORDERING=1

6. Restart the compute node.

7. Check whether the configuration is successful.

   #mst start

   #mst status

   #mlxconfig -y -d /dev/mst/mt4119_pciconf0 q |grep PCI_WR

   

Check After Recovery

1. Multiple compute nodes can ping each other to ensure that the IB network is reachable.

2. Run the ib_send_bw command to test the IB channel link again.

Suggestion and Summary

N/A

Applicable Versions

OceanStor 100D 8.0.3; OceanStor Pacific series 8.1.0