Got it

What's the difference between wide and narrow dependencies?

Created: Apr 8, 2022 11:32:08Latest reply: Jun 24, 2022 03:37:57 325 2 1 0 0
  HiCoins as reward: 0 (problem unresolved)

Hello, everyone!

Do you know the difference between wide and narrow dependencies?

Can you describe it simply?

  • x
  • convention:

Featured Answers
olive.zhao
Admin Created Apr 8, 2022 11:35:54

Hello, friend!

Narrow dependency indicates that each partition of a parent RDD can be used by at most one partition of a child RDD, for example, map, filter, and union.

Wide dependency indicates that the partitions of multiple child RDDs depend on the partition of the same parent RDD, for example, groupByKey, reduceByKey, and sortByKey.

If a node is faulty:

  • Narrow dependency: Only the parent RDD partition corresponding to the child RDD partition needs to be recalculated.

  • Wide dependency: In extreme cases, all parent RDD partitions need to be recalculated.

As shown in the following figure, if the b1 partition is lost, a1, a2, and a3 need to be recalculated.

Fault Tolerance

Data Transmission

  • Wide dependency usually corresponds to shuffle operations. During the running process, the partition of the same parent RDD needs to be transferred to different child RDD partitions, which may involve data transmission between multiple nodes.

  • The partition of each parent RDD on which narrow dependency exists is transferred to only one child RDD partition. Generally, the conversion can be completed on one node.

Hope this helps!

View more
  • x
  • convention:

All Answers
olive.zhao
olive.zhao Admin Created Apr 8, 2022 11:35:54

Hello, friend!

Narrow dependency indicates that each partition of a parent RDD can be used by at most one partition of a child RDD, for example, map, filter, and union.

Wide dependency indicates that the partitions of multiple child RDDs depend on the partition of the same parent RDD, for example, groupByKey, reduceByKey, and sortByKey.

If a node is faulty:

  • Narrow dependency: Only the parent RDD partition corresponding to the child RDD partition needs to be recalculated.

  • Wide dependency: In extreme cases, all parent RDD partitions need to be recalculated.

As shown in the following figure, if the b1 partition is lost, a1, a2, and a3 need to be recalculated.

Fault Tolerance

Data Transmission

  • Wide dependency usually corresponds to shuffle operations. During the running process, the partition of the same parent RDD needs to be transferred to different child RDD partitions, which may involve data transmission between multiple nodes.

  • The partition of each parent RDD on which narrow dependency exists is transferred to only one child RDD partition. Generally, the conversion can be completed on one node.

Hope this helps!

View more
  • x
  • convention:

ThandekaWendy
ThandekaWendy Created Jun 24, 2022 03:37:57

Narrow dependency indicates that each partition of a parent RDD can be used by at most one partition of a child RDD, for example, map, filter, and union. Wide dependency indicates that the partitions of multiple child RDDs depend on the partition of the same parent RDD, for example, groupByKey, reduceByKey, and sortByKey.
View more
  • x
  • convention:

Comment

You need to log in to comment to the post Login | Register
Comment

Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:
  • Politically sensitive content
  • Content concerning pornography, gambling, and drug abuse
  • Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy
Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " User Agreement."

My Followers

Login and enjoy all the member benefits

Login

Block
Are you sure to block this user?
Users on your blacklist cannot comment on your post,cannot mention you, cannot send you private messages.
Reminder
Please bind your phone number to obtain invitation bonus.
Information Protection Guide
Thanks for using Huawei Enterprise Support Community! We will help you learn how we collect, use, store and share your personal information and the rights you have in accordance with Privacy Policy and User Agreement.