Spark tasks are slow in coarse-grained computing. When Spark invokes the getsplit method, a large number of small files are generated.

Created: Jun 30, 2019 08:02:38 174 0 11 0 0

Symptom

The Spark task takes a long time to calculate daily SDRs. Due to the increase in the number of components in the cluster capacity expansion, the mapreduce.input.fileinputformat.list-status.num-threads optimization parameters in the product do not improve the performance on the FI side. Therefore, the cause must be analyzed.

Solution

The getMoreSplits method does not have optimization parameters. The speed of traversing files is related to the number of files. Therefore, you are advised to reduce the number of files on the service side to accelerate the running.

x
convention：
Emotion(0)

Like (11) Dislike (0) Favorite(0) Report

 Previous: When Spark tasks are running, a large number of executor are in the dead state.

Next: The Spark Streaming application reports an error



Comment

Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:

Politically sensitive content
Content concerning pornography, gambling, and drug abuse
Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy

Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " User Agreement."

My Followers

Are you sure to block this user?
Users on your blacklist cannot comment on your post,cannot mention you, cannot send you private messages.

Please bind your phone number to obtain invitation bonus.

Enterprise

Consumer

Corporate

Carriers

Spark tasks are slow in coarse-grained computing. When Spark invokes the getsplit method, a large number of small files are generated.

Third Party’s Trade Secret

My Followers