Got it

Kafka Consumption Rebalance Error

Latest reply: Feb 28, 2022 13:47:27 906 1 0 0 0

Hello, everyone! 

I will share with you how to deal with the problem of consumption rebalance errors when using Kafka.

Context and Symptom

The rebalance exception occurred every 2 to 3 minutes. After processing a batch of poll messages, the consumer submitted the offset to the Broker. An error was reported.

08-09 11:01:11 131 pool-7-thread-3 ERROR [] -commit failedorg.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.sendOffsetCommitRequest(ConsumerCoordinator.java:713) ~[MsgAgent-jar-with-dependencies.jar:na]at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:596) ~[MsgAgent-jar-with-dependencies.jar:na]at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1218) ~[MsgAgent-jar-with-dependencies.jar:na]at com.today.eventbus.common.MsgConsumer.run(MsgConsumer.java:121) ~[MsgAgent-jar-with-dependencies.jar:na]at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_161]at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_161]at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]

Cause Analysis

The poll() method returned a message set in polling mode. A batch of messages can be obtained by one invoke.

kafkaConsumer invoked the polling method only once to obtain the messages. To continuously obtain messages, the client used an external loop to continuously invoke the polling method of the consumer. The next poll was performed only after all messages in a batch were processed. The server has specified the maximum interval between two polling operations (max.poll.interval.ms). If the result returned in a polling operation cannot be processed in time, the client took more time to process a batch of messages than the maximum interval. In this case, the server may remove the client and trigger rebalance.

Solution

  1. Increase max.poll.interval.ms. The default interval for Kafka consumers is 300s.

    max.poll.interval.ms=300

  2. Sets the partition obtaining threshold.

    kafkaConsumer invokes the polling method only once to obtain the messages. To continuously obtain messages, the client uses an external loop to continuously invoke poll(). The next poll is performed only after all messages in a batch are processed.

    max.poll.records = 50

  3. After a message obtained from polling is processed, a new message is submitted. If a message fails to be submitted, the loop exits immediately. In this case, the Kafka performs rebalance and continues to consume the message from the current offset.

    while (isRunning) {
        ConsumerRecords<KEY, VALUE> records = consumer.poll(100);
        if (records != null && records.count() > 0) {
    
             for (ConsumerRecord<KEY, VALUE> record : records) {
                 dealMessage(bizConsumer, record.value());
                 try {
                      //Submit after all records are complete.
                      consumer.commitSync();
                 } catch (CommitFailedException e) {
                        logger.error("commit failed,will break this for loop", e);
                        break;
                 }
             }
    }

We warmly welcome you to enjoy our community!

  • x
  • convention:

olive.zhao
Admin Created Feb 28, 2022 13:47:27

  • x
  • convention:

Comment

You need to log in to comment to the post Login | Register
Comment

Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:
  • Politically sensitive content
  • Content concerning pornography, gambling, and drug abuse
  • Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy
Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " User Agreement."

My Followers

Login and enjoy all the member benefits

Login

Block
Are you sure to block this user?
Users on your blacklist cannot comment on your post,cannot mention you, cannot send you private messages.
Reminder
Please bind your phone number to obtain invitation bonus.
Information Protection Guide
Thanks for using Huawei Enterprise Support Community! We will help you learn how we collect, use, store and share your personal information and the rights you have in accordance with Privacy Policy and User Agreement.