Poor Performance of Uploading Small Objects in a Newly Deployed Environment

Problem Information

Problem Symptom

When a small object is uploaded for the first time to a newly deployed environment, TPS is low.

Problem Diagnosis

1. When a system is newly deployed, the index layer creates only one chunk for each table. As a result, the small object upload performance depends on the processing capabilities of a single chunk.

2. Run the following commands on any access node to check the number of chunks in the object table:

   mongoshell clientuse configdb.chunks.find({"ns":"dfv_s3_object_service.object"}).count()

   In the example execution result shown in the following figure, there are 32 chunks in the object table. If only one chunk is displayed, adjust the split threshold of the chunk in the object table by referring to the Solution to split the chunk into multiple chunks quickly.

Causes

When a system is newly deployed, the index layer creates only one chunk for each table. As a result, the small object upload performance depends on the processing capabilities of a single chunk, and the performance of the entire system is poor.

Solution

1. Run the following command to log in to the mongo client:

   mongoshell client

2. Run the following commands to adjust the split threshold of the mongodb chunk:

   use configdb.settings.save( { _id:"chunksize", value: 10 } );db.settings.update({_id : "balancer"}, {\$set : {useTPSForSplit : false}}, {upsert : true})db.settings.update({_id : "balancer"}, {\$set: {chunkReuseTimerSec : 30}}, {upsert : true})

3. After the split threshold is adjusted, you can continue to upload small objects and wait for splitting. Then, run the following commands to check the number of chunks in the object table:

   use configdb.chunks.find({"ns":"dfv_s3_object_service.object"}).count()

4. The recommended number of chunks is 16 for 3 or 4 nodes, 32 for 5 to 7 nodes, and 64 for 8 or 9 nodes. When the number of chunks reaches the recommended value, run the following commands to change the split threshold of the chunk to the default value to avoid an excessive number of chunks.

   use configdb.settings.update({_id : "balancer"}, {\$set: {tpsSplitThreshold : 800}}, {upsert : true})db.settings.update({_id : "balancer"}, {\$set : {useTPSForSplit : true}}, {upsert : true})db.settings.update({_id : "balancer"}, {\$set: {chunkReuseTimerSec : 600}}, {upsert : true})db.settings.save( { _id:"chunksize", value: 20480 } );

Check After Recovery

After the problem is resolved as instructed in the solution, check whether the performance of uploading small objects is improved.

Suggestion and Summary

N/A

Applicable Versions

All