Got it

Producer Occasionally Fails to Send Data and "Too many open files in system" Is Displayed in Logs

Latest reply: Mar 18, 2022 09:06:29 2375 5 5 0 0

Hello, everyone! 

I will share with you how to deal with the problem  "Too many open files in system" when using Kafka.

Symptom

When Producer is used to send data to Kafka, the client fails to send data.

Producer fails to send data

Possible Causes

  • The Kafka service is abnormal.

  • The network is abnormal.

  • The Kafka topic is abnormal.

  • Operating system parameters are missing.

Fault Locating

1. Log in to FusionInsight Manager, choose Services > Kafka, and check the Kafka cluster status. The check result indicates that the Kafka cluster status is good, and monitoring indicators are correctly displayed.

2. View the error topic information in the SparkStreaming log.

Run the Kafka commands to obtain the topic assignment information and replica synchronization information, and check the result.

kafka-topics.sh --describe --zookeeper <zk_host:port/chroot>

If information in the following figure is displayed, the topic is normal. All partitions have normal leader information.

Topic status

3. Run the following ping commands to check whether Kafka can be connected.

ping kafka service IP address

telnet kafka service IP address kafka service port

If the ping or telnet operation fails, contact the network team to rectify the fault.

4. Log in to the Kafka Broker using SSH.

Run the cd /var/log/Bigdata/kafka/broker command to go to the log directory.

Check on server.log indicates that the error message is displayed in the log shown in the following figure.

Log exception

5. Output of the lsof command used to check the handle usage of the Kafka process on the current node shows that the number of handles used by the Kafka process reaches 470,000.

Handles

6. Check on service codes indicates that the Producer object is frequently created and therefore is not closed normally.

If the number of operation handles used after step 6 is small, run the ulimit -n command to check whether the number of operation handles in the cluster is 640,000.

If yes, run the ps -ef | grep kafka command to find the Kafka producer ID (pid) and check the number of operation handles of the Kafka process, and run the

cat /proc/pid/limits command to check whether the maximum number of handles of the Kafka process is the same as the value of ulimit -n in the preceding step. 

Solution

  1. Stop the current application to ensure that the handles on the server do not increase.

  2. Optimize the application code to resolve the handle leakage problem. You are advised to use a Producer object globally. After the object is used, invoke the close interface to close the handle.


We warmly welcome you to enjoy our community!

  • x
  • convention:

olive.zhao
Admin Created Mar 18, 2022 09:06:29

Do you know how to reset gandalf password?

Don't worry, you can refer How to reset gandalf password? to reset gandalf password.

View more
  • x
  • convention:

olive.zhao
Admin Created Jan 12, 2022 08:56:30

Thanks for your sharing!
View more
  • x
  • convention:

little_fish
Admin Created Jan 12, 2022 08:59:08

How to handle Too many open files in system for Kafka.
View more
  • x
  • convention:

NTan33
Created Jan 19, 2022 02:18:23

A good troubleshooting example to reference.
View more
  • x
  • convention:

user_4358465
Created Jan 19, 2022 10:08:49

Excellent solution post!
View more
  • x
  • convention:

olive.zhao
Admin Created Mar 18, 2022 09:06:29

Do you know how to reset gandalf password?

Don't worry, you can refer How to reset gandalf password? to reset gandalf password.

View more
  • x
  • convention:

Comment

You need to log in to comment to the post Login | Register
Comment

Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:
  • Politically sensitive content
  • Content concerning pornography, gambling, and drug abuse
  • Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy
Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " User Agreement."

My Followers

Login and enjoy all the member benefits

Login

Block
Are you sure to block this user?
Users on your blacklist cannot comment on your post,cannot mention you, cannot send you private messages.
Reminder
Please bind your phone number to obtain invitation bonus.
Information Protection Guide
Thanks for using Huawei Enterprise Support Community! We will help you learn how we collect, use, store and share your personal information and the rights you have in accordance with Privacy Policy and User Agreement.