When Spark tasks are running, a large number of executor are in the dead state.

Created: Jun 30, 2019 08:03:24 2461 0 11 0 0

Symptom

When Spark tasks are running, a large number of executor status is Dead, and some error information is displayed in task logs.

Solution

The memory is insufficient. As a result, the executor is killed and the spark.executor.memory memory is adjusted.

x
convention：
Emotion(0)

Like (11) Dislike (0) Favorite(0) Report

 Previous: The number of reported data groups exceeds the threshold

Next: Spark tasks are slow in coarse-grained computing. When Spark invokes the getsplit method, a large number of small files are generated.



Comment

Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:

Politically sensitive content
Content concerning pornography, gambling, and drug abuse
Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy

Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " User Agreement."

My Followers

Are you sure to block this user?
Users on your blacklist cannot comment on your post,cannot mention you, cannot send you private messages.

Please bind your phone number to obtain invitation bonus.

Enterprise

Consumer

Corporate

Carriers

When Spark tasks are running, a large number of executor are in the dead state.

Third Party’s Trade Secret

My Followers