NAS service performance suddenly drops

168 0 12 12

NAS service performance suddenly drops. A customer has two FC ******rvice hosts (running Red Hat operating systems) connected to a storage array using a Fibre Channel switch and two NAS hosts (running Red Hat operating systems) connected to the same storage array using a GE switch

The storage array is configured with four LUNs, 1 TB each, with two controllers A and two controllers B and two file systems, with one controller A and one controller B

Service model: SAN+NAS; SAN: OLTP; NAS: sequential 1 MB 100% write

LUN services are started first, then NAS services on host A, and then NAS services on host B three minutes later.

After a while, host A's bandwidth drops from approximately 110 MB/s to approximately 5 MB/s. The resp on vdbench is 400+ ms, and the corresponding file system's I/O latency on the DeviceManager is 160+ ms.

This issue is a known issue and has been resolved in V300R002C10. The root cause is as follows:

The service pressure on NL-SAS disks keeps increasing due to NAS+******rvices, and FC ******rvices keep snatching resources. Therefore, the internal traffic control policy in the storage system restricts the sequential NAS read and write services, and NAS performance drops significantly.

1. Check whether the storage system version is earlier than V300R002C10. If yes, go to the next step.

2. Check whether logs contain information about traffic control and whether the traffic control policy is effective during the issue occurrence. If yes, go to the next step.

3. Use the performance statistics function of the DeviceManager to check whether back-end disks have reached their performance bottleneck. If yes, go to the next step.

4. Check whether SAN and NAS services run concurrently and whether only NAS services encounter the performance drop. If yes, the issue is confirmed.

Workaround 1

NL-SAS disks are not recommended for database services. Therefore, advise the customer to add SAS disks based on the database services' needs. Assign all SSDs and SAS disks to the same disk domain and enable the database services (FC SAN) to run based on SSDs and SAS disks. Assign all NL-SAS disks to the same disk domain and enable NAS file sharing services (NAS) to run based on NL-SAS disks.

Advantages: Services are separated and do not affect each other in terms of performance; impacts of faults are restricted to a certain range (such as a disk domain).
Disadvantage: The cost is high.

Workaround 2

In preliminary delivery, storage configuration is planned based on NAS file sharing and VM services, and all disks are assigned to the same disk domain. Therefore, if database services are to be added, advise the customer to re-assign disks.

Allocate a disk domain to database services (FC SAN). The disk domain consists of SSDs and some NL-SAS disks. In addition, enable the SmartTier feature. Use SSDs for data LUNs and NL-SAS disks for data backup LUNs of the database.

Allocate a disk domain to NAS file sharing services (FC SAN). The disk domain consists solely of NL-SAS disks.

Advantages: Services are separated and do not affect each other in terms of performance; impacts of faults are restricted to a certain range (such as a disk domain).
Disadvantages: Large changes have to be implemented on the current configuration; fewer disks are allocated to a certain service; FC SAN performance will drop if the space provided by SSDs is insufficient.

Workaround 3

Leave current configuration unchanged (one disk domain), and refine the LUN division (because database services are added).

Put all space from SSDs and a small fraction of space from NL-SAS disks to a resource pool for storing data LUNs of the database. Enable SmartTier for performance improvement (if NL-SAS space is used). (If SSDs provide sufficient space, you do not need NL-SAS disks.)

Remaining space from NL-SAS disks is used to store data backup LUNs of the database (NAS file sharing).

Advantage: Little change to the current configuration is required.
Disadvantages: You need to know clearly the customer's services; if SSDs fail to provide sufficient space, the performance of FC SAN and NAS services will be affected, and the performance drop of the services in the test will recur.


  • x
  • convention:

Comment

You need to log in to reply to the post Login | Register

Notice Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:
  • Politically sensitive content
  • Content concerning pornography, gambling, and drug abuse
  • Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy
Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " Privacy."
If the attachment button is not available, update the Adobe Flash Player to the latest version!
Login and enjoy all the member benefits

Login and enjoy all the member benefits

Login