Hello, everyone!
I will share with you how to deal with the problem of production performance to deteriorate when using Kafka.
Symptom
After the open source Kafka client librdkafka of C++ is used, the production performance is low. After the iftop test tool is used, a large amount of metadata is written back.
Cause Analysis
By default, the Kafka client of the C++ version returns the metadata of all topics in the cluster to the producer. If the number of topics in the cluster is large and the bandwidth is insufficient, this problem occurs.
Solution
The official reply is as follows:
On the producer and consumer, set topic.metadata.refresh.sparse to true to ensure that only the required topic information is retained for each writeback. In addition, reduce the frequency of obtaining metadata
by increasing the value of topic.metadata.refresh.interval.ms.
We warmly welcome you to enjoy our community!
