Got it

Failed to Start Up Kafka Node

Created: May 17, 2022 09:32:56Latest reply: May 17, 2022 09:41:54 294 4 0 0 0
  Rewarded HiCoins: 0 (problem resolved)

Hello Huawei engineers!


This post enquires about the issue that I failed to start up the Kafka node. Please see below for more details.


ISSUE DESCRIPTION


I use the Huawei FusionInsight solution and I can't start up the Kafka node. Could anyone help me?


Thanks in advance!

  • x
  • convention:

Featured Answers
olive.zhao
Admin Created May 17, 2022 09:40:57

Posted by Jackson.F at 2022-05-17 09:37 Hello, Olive!The FusionInsight version is 6.5.1.The cluster is powered off unexpectedly because of t ...

Hello, Jackson!

After the cluster is powered off unexpectedly and then powered on, some Broker nodes fail to start. The AL-12007 progress fault alarm may be generated for the cluster. For a small cluster (with three or fewer Broker nodes), the ALM-38006 alarm may be generated indicating that the percentage of partitions that are not completely synchronized exceeds the threshold. Check the server.log file of the Broker node that fails to be started. The default path of this log file is /var/log/Bigdata/kafka/broker/server.log. The following error information is displayed (the marked area indicates the damaged file):

kafka node

The possible cause is that the Kafka metadata storage file is damaged due to unexpected power failure. As a result, Broker fails to start. In the preceding figure, the damaged file is the leader-epoch-checkpoint metadata file of a partition in the Kafka data storage directory.

You can backup the damaged file and restart the Kafka service.

1. Back up the damaged file to another non-Kafka data storage directory (specified by log.dirs).

2. Delete the damaged file and restart the Kafka service. If the restart fails, view the server.log file to check whether other files contain the same error information. If yes, more than one file is damaged. In this case, repeat Step 1-Step 2.

3. If the restart is successful, check whether the Kafka service is recovered. After the service runs normally for a period of time, you can delete the backup files. If the restart fails, contact Kafka O&M personnel.

Hope this helps!

View more
  • x
  • convention:

All Answers
olive.zhao
olive.zhao Admin Created May 17, 2022 09:34:32

Hello, Jackson!

Which FusionInsight version do you use?

What is the fault that occurred before the Kafka node was started?

Why is the Kafka node power off?

Wait for your feedback!

View more
  • x
  • convention:

Jackson.F
Jackson.F Created May 17, 2022 09:37:04

Hello, Olive!

The FusionInsight version is 6.5.1.

The cluster is powered off unexpectedly because of the line problem in the equipment room.

View more
  • x
  • convention:

olive.zhao
olive.zhao Admin Created May 17, 2022 09:40:57

Posted by Jackson.F at 2022-05-17 09:37 Hello, Olive!The FusionInsight version is 6.5.1.The cluster is powered off unexpectedly because of t ...

Hello, Jackson!

After the cluster is powered off unexpectedly and then powered on, some Broker nodes fail to start. The AL-12007 progress fault alarm may be generated for the cluster. For a small cluster (with three or fewer Broker nodes), the ALM-38006 alarm may be generated indicating that the percentage of partitions that are not completely synchronized exceeds the threshold. Check the server.log file of the Broker node that fails to be started. The default path of this log file is /var/log/Bigdata/kafka/broker/server.log. The following error information is displayed (the marked area indicates the damaged file):

kafka node

The possible cause is that the Kafka metadata storage file is damaged due to unexpected power failure. As a result, Broker fails to start. In the preceding figure, the damaged file is the leader-epoch-checkpoint metadata file of a partition in the Kafka data storage directory.

You can backup the damaged file and restart the Kafka service.

1. Back up the damaged file to another non-Kafka data storage directory (specified by log.dirs).

2. Delete the damaged file and restart the Kafka service. If the restart fails, view the server.log file to check whether other files contain the same error information. If yes, more than one file is damaged. In this case, repeat Step 1-Step 2.

3. If the restart is successful, check whether the Kafka service is recovered. After the service runs normally for a period of time, you can delete the backup files. If the restart fails, contact Kafka O&M personnel.

Hope this helps!

View more
  • x
  • convention:

Jackson.F
Jackson.F Created May 17, 2022 09:41:54

Posted by olive.zhao at 2022-05-17 09:40 Hello, Jackson!After the cluster is powered off unexpectedly and then powered on, some Broker node ...
Thanks, Olive!
I will try it.
View more
  • x
  • convention:

Comment

You need to log in to comment to the post Login | Register
Comment

Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:
  • Politically sensitive content
  • Content concerning pornography, gambling, and drug abuse
  • Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy
Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " User Agreement."

My Followers

Login and enjoy all the member benefits

Login

Block
Are you sure to block this user?
Users on your blacklist cannot comment on your post,cannot mention you, cannot send you private messages.
Reminder
Please bind your phone number to obtain invitation bonus.
Information Protection Guide
Thanks for using Huawei Enterprise Support Community! We will help you learn how we collect, use, store and share your personal information and the rights you have in accordance with Privacy Policy and User Agreement.