Poor Performance in New Buckets When Multiple Buckets Exist in a System

Problem Information

Problem Symptom

When multiple buckets exist in a system, the performance of new buckets is poor.

Problem Diagnosis

1. Log in to any node and run the mongoshell client command.

   mongoshell clientuse config

2. Check the number of chunks of buckets with poor performance.

   db.chunks.find({"ns":"dfv_s3_object_service.object", "min.bn":"bucket-dfv-test-01"})

   

   If the number of chunks is less than three, the number of chunks of the bucket is too small, putting overloaded pressure on a single chunk. As a result, the service performance of the bucket is poor.

Causes

Multiple buckets exist in the system. Because data is written to the existing buckets, a large number of chunks are occupied by the buckets in the system. As a result, when the number of chunks reaches the threshold, the chunks can hardly be split. Data in new buckets is stored in one chunk, causing a hotspot chunk. The overloaded pressure deteriorates the performance of the single chunk.

Solution

1. Check whether the total number of chunks in the cluster is equal to 40 x Number of nodes in the cluster.

   mongoshell clientuse configdb.chunks.find({status:1}).count()

• If yes, the number of chunks in the cluster reaches the threshold. In this case, merge some chunks and then go to 2 to split chunks.

• If the number of chunks is at least 10 less than 40 x Number of nodes in the cluster, go to 3 to directly split chunks.

Merge chunks and then split them.

cd /opt/dfv/index_layer/config/27015/action/

sh selectMergeCandidates.sh dfv_s3_object_service.object

dfv_s3_object_service.object.output

db.rebalanceevents.find()

Run the following commands to merge the two adjacent chunks with the smallest sizes:

mongoshell client

use config

db.adminCommand({ mergeChunks : "dfv_s3_object_service.object", chunkid : [ "4f", "45e"] })

The execution of the preceding commands takes a long time. After the commands are executed, re-check the distribution of chunks.

cd /opt/dfv/index_layer/config/27015/action/sh selectMergeCandidates.sh dfv_s3_object_service.objectdfv_s3_object_service.object.output

Repeat this step until the number of chunks in the cluster decreases by 5.

Log in to any node and run the mongoshell client command to observe prefix naming rules of new buckets, for example, 0f000fa1_obs_test.

mongoshell clientuse configdb.adminCommand({split:"dfv_s3_object_service.object", manualsplit:true,find:{bn:"obs_test",on:"xxx"}})

Manually split chunks until the performance of buckets is improved significantly.

Check After Recovery

After handling the issue based on Solution, check whether the performance of new buckets is improved.

Suggestion and Summary

N/A

Applicable Versions

All