Got it
Huawei Enterprise Support Community
Community Forums Cloud & Big Data
Spark Task Fails to Be Ex...

Spark Task Fails to Be Executed After a Yarn Active/Standby Switchover

Created: Nov 27, 2020 13:02:50Latest reply: Nov 27, 2020 13:07:32 241 4 0 0 0
  HiCoins as reward： 0 (problem unresolved)
View the author 1#

Hi team,

When I run Spark applications, HDFS write operations are performed. During the process, the following error occurs:

Exception in thread "Driver" org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory
hdfs://hacluster/testsave already exists
at org.apache.hadoop.mapred.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:132)
at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopDataset(PairRDDFunctions.scala:1021)
at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:931)
at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:840)
at org.apache.spark.rdd.RDD.saveAsTextFile(RDD.scala:1177)
at com.spark.test.wordcountWithSave$.main(wordcountWithSave.scala:30)
at com.spark.test.wordcountWithSave.main(wordcountWithSave.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:460)

Please help me analyze the cause of the error. Thank you.

Like (0) Dislike (0) Favorite(0) Share Report
Previous: How do I replace CPU?
Next: Not attempting to re-login since the last re-login was attempted less than 600 seconds before
Featured Answers

Recommended answer

(0) (0)
little_fish
Admin Created Nov 27, 2020 13:07:32

Dear Axe,

Perform the following operations to identify the second cause:


1. On FusionInsight Manager, check whether the active and standby nodes are changed. If yes, the fault is caused by the active/standby switchover.


2. View Yarn logs. If the following information is displayed, the error is caused by the active/standby switchover.


Checking for any old active which needs to be fenced...

No old node to fence

Writing znode /yarn-leader-election/yarncluster/ActiveBreadCrumb to indicate that the local node is the most recent active...

found resource yarn-site.xml at file:/opt/huawei/Bigdata/FusionInsight/etc/1_7_ResourceManager/yarn-site.xml

USER=mapredOPERATION=refreshAdminAclsTARGET=AdminServiceRESULT=SUCCESS

Transitioning to active state


  • Cause: The HDFS directory already exists.


Modify the HDFS directory defined in the application or delete the existing directory in the HDFS.


  • Cause: A Yarn active/standby switchover is performed.


There are two methods to solve this problem:


  • Delete the existing directory or modify the directory of the application, and then submit the application again.


  • On the client, open the $Spark_Client/conf/spark-defaults.conf file and set spark.hadoop.validateOutputSpecs to false.

Thank you.

View more
  • x
  • convention：
All Answers
(1) (0)
2#
little_fish
little_fish Admin Created Nov 27, 2020 13:04:04

Dear,
Here are 2 causes.
Cause 1: Before the Spark application is executed, a directory for writing data to the application exists in HDFS.
Cause 2: After the Spark application is executed, the application writes data to the specified directory in the HDFS, and the Yarn active/standby switchover occurs. As a result, the application is executed again, and this error occurs.
Thank you.
View more
  • x
  • convention：

Axe_Z
Axe_Z Created Nov 27, 2020 13:04:43 (0) (0)
What can I do if it's cause 2?  
little_fish
little_fish Reply Axe_Z  Created Nov 28, 2020 00:44:55 (0) (0)
 
(0) (0)
3#
little_fish
little_fish Admin Created Nov 27, 2020 13:07:32

Dear Axe,

Perform the following operations to identify the second cause:


1. On FusionInsight Manager, check whether the active and standby nodes are changed. If yes, the fault is caused by the active/standby switchover.


2. View Yarn logs. If the following information is displayed, the error is caused by the active/standby switchover.


Checking for any old active which needs to be fenced...

No old node to fence

Writing znode /yarn-leader-election/yarncluster/ActiveBreadCrumb to indicate that the local node is the most recent active...

found resource yarn-site.xml at file:/opt/huawei/Bigdata/FusionInsight/etc/1_7_ResourceManager/yarn-site.xml

USER=mapredOPERATION=refreshAdminAclsTARGET=AdminServiceRESULT=SUCCESS

Transitioning to active state


  • Cause: The HDFS directory already exists.


Modify the HDFS directory defined in the application or delete the existing directory in the HDFS.


  • Cause: A Yarn active/standby switchover is performed.


There are two methods to solve this problem:


  • Delete the existing directory or modify the directory of the application, and then submit the application again.


  • On the client, open the $Spark_Client/conf/spark-defaults.conf file and set spark.hadoop.validateOutputSpecs to false.

Thank you.

View more
  • x
  • convention：
Back to list

Comment

Advanced
B Color Image Link Quote Code Smilies
You need to log in to comment to the post Login | Register
Comment

Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:
  • Politically sensitive content
  • Content concerning pornography, gambling, and drug abuse
  • Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy
Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " User Agreement."

My Followers

Login and enjoy all the member benefits

Login
Huawei Website Support About Huawei Privacy User Agreement Terms of Use Cookies Cookie Settings
Huawei Enterprise Huawei Carrier Forum Training & Certification

Contact Us: e_online@huawei.com Copyright © 2022 Huawei Technologies Co., Ltd. All rights reserved.

APP

Block
Are you sure to block this user?
Users on your blacklist cannot comment on your post,cannot mention you, cannot send you private messages.
Reminder
Please bind your phone number to obtain invitation bonus.