Got it

Address already in use: Service 'SparkUI' failed after 16 retries!

Created: Nov 27, 2020 12:53:08Latest reply: Jan 29, 2022 01:26:31 2049 6 1 0 0
  HiCoins as reward: 0 (problem unresolved)

Hi team,

The following error occurred while submitting the task. This problem occurs when multiple tasks are submitted at the same time. What should I do? Thank you.

2014-09-17 07:27,597 INFO [main] Successfully started service 'HTTP file server' on port 23503. org.apache.spark.Logging$class.logInfo(Logging.scala:59)
2014-09-17 07:27,875 WARN [main] Service 'SparkUI' could not bind on port 23000. Attempting port 23001. org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2014-09-17 07:27,942 WARN [main] Service 'SparkUI' could not bind on port 23001. Attempting port 23002. org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2014-09-17 07:27,003 WARN [main] Service 'SparkUI' could not bind on port 23002. Attempting port 23003. org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2014-09-17 07:27,086 WARN [main] Service 'SparkUI' could not bind on port 23003. Attempting port 23004. org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2014-09-17 07:27,147 WARN [main] Service 'SparkUI' could not bind on port 23004. Attempting port 23005. org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2014-09-17 07:27,213 WARN [main] Service 'SparkUI' could not bind on port 23005. Attempting port 23006. org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2014-09-17 07:27,275 WARN [main] Service 'SparkUI' could not bind on port 23006. Attempting port 23007. org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2014-09-17 07:27,336 WARN [main] Service 'SparkUI' could not bind on port 23007. Attempting port 23008. org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2014-09-17 07:27,396 WARN [main] Service 'SparkUI' could not bind on port 23008. Attempting port 23009. org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2014-09-17 07:27,461 WARN [main] Service 'SparkUI' could not bind on port 23009. Attempting port 23010. org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2014-09-17 07:27,523 WARN [main] Service 'SparkUI' could not bind on port 23010. Attempting port 23011. org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2014-09-17 07:27,586 WARN [main] Service 'SparkUI' could not bind on port 23011. Attempting port 23012. org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2014-09-17 07:27,644 WARN [main] Service 'SparkUI' could not bind on port 23012. Attempting port 23013. org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2014-09-17 07:27,704 WARN [main] Service 'SparkUI' could not bind on port 23013. Attempting port 23014. org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2014-09-17 07:27,763 WARN [main] Service 'SparkUI' could not bind on port 23014. Attempting port 23015. org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2014-09-17 07:27,825 WARN [main] Service 'SparkUI' could not bind on port 23015. Attempting port 23016. org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2014-09-17 07:27,887 ERROR [main] Failed to bind SparkUI org.apache.spark.Logging$class.logError(Logging.scala:96)
java.net.BindException: Address already in use: Service 'SparkUI' failed after 16 retries!
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:444)
at sun.nio.ch.Net.bind(Net.java:436)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.eclipse.jetty.server.Server.doStart(Server.java:293)
¡¡


  • x
  • convention:

Featured Answers
little_fish
Admin Created Nov 27, 2020 12:58:10

Dear Axe,

Run the jps command to check the number of tasks submitted on the current node. If the number of tasks submitted on the current node exceeds 16, this error occurs.


Perform any of the following steps to solve the problem:


  • When initializing SparkConf, add the conf.set("spark.port.maxRetries, "100") statement.


  • When using spark-submit to submit a task, add --conf spark.port.maxRetries=100 to the command line.


  • Add spark.port.maxRetries 100 to spark-defaults.conf.


You can replace 100 with any number you want. The larger the number, the more tasks you can run at the same time.

Thank you.

View more
  • x
  • convention:

Pekkle
Pekkle Created Jan 28, 2022 04:30:42 (0) (0)
Hi, little_fish, I am using Spark 2.4.5 on Hadoop Yarn. I can see its default spark.ui.port is 4040 from official website (but not 23000). While I add this config into my jobs to point it to other port number, it doesn't work at all. It appears to be a random number of every single Spark job. Even if I just spin up only one job in the whole cluster, it is still a random one rather than my configured one. Do you know what is the issue please? Thanks.  
little_fish
little_fish Reply Pekkle  Created Jan 29, 2022 01:23:42 (0) (0)
4040 is for web UI  
All Answers
little_fish
little_fish Admin Created Nov 27, 2020 12:54:19

Dear,
Each Spark task starts a Driver port, that is, SparkUI. The default port is 23000. If the port is occupied, the system randomly selects a port for retry. By default, the system will retry for 16 times. If 16 retry attempts fail, the task will be stopped. Thank you.
View more
  • x
  • convention:

Axe_Z
Axe_Z Created Nov 27, 2020 12:55:33

Posted by little_fish at 2020-11-27 12:54 Dear,Each Spark task starts a Driver port, that is, SparkUI. The default port is 23000. If the port ...
How do I config SparkConf? Thank you.
View more
  • x
  • convention:

little_fish
little_fish Admin Created Nov 27, 2020 12:58:10

Dear Axe,

Run the jps command to check the number of tasks submitted on the current node. If the number of tasks submitted on the current node exceeds 16, this error occurs.


Perform any of the following steps to solve the problem:


  • When initializing SparkConf, add the conf.set("spark.port.maxRetries, "100") statement.


  • When using spark-submit to submit a task, add --conf spark.port.maxRetries=100 to the command line.


  • Add spark.port.maxRetries 100 to spark-defaults.conf.


You can replace 100 with any number you want. The larger the number, the more tasks you can run at the same time.

Thank you.

View more
  • x
  • convention:

Pekkle
Pekkle Created Jan 28, 2022 04:30:42 (0) (0)
Hi, little_fish, I am using Spark 2.4.5 on Hadoop Yarn. I can see its default spark.ui.port is 4040 from official website (but not 23000). While I add this config into my jobs to point it to other port number, it doesn't work at all. It appears to be a random number of every single Spark job. Even if I just spin up only one job in the whole cluster, it is still a random one rather than my configured one. Do you know what is the issue please? Thanks.  
little_fish
little_fish Reply Pekkle  Created Jan 29, 2022 01:23:42 (0) (0)
4040 is for web UI  
little_fish
little_fish Admin Created Jan 29, 2022 01:26:31

Dear @Pekkle ,


--conf spark.ui.port=4050 is to change the port Spark feature. You can set it in your codes, such as:

val conf = new SparkConf().setAppName(s"SimpleApp").set("spark.ui.port", "4050")
val sc = new SparkContext(conf)


If you launch two applications with the same value for "spark.ui.port" the second app will fail to bind to 4050 and try 4051.

Refer to https://spark.apache.org/docs/2.4.5/security.html#web-ui


Thank you.

View more
  • x
  • convention:

Comment

You need to log in to comment to the post Login | Register
Comment

Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:
  • Politically sensitive content
  • Content concerning pornography, gambling, and drug abuse
  • Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy
Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " User Agreement."

My Followers

Login and enjoy all the member benefits

Login

Block
Are you sure to block this user?
Users on your blacklist cannot comment on your post,cannot mention you, cannot send you private messages.
Reminder
Please bind your phone number to obtain invitation bonus.
Information Protection Guide
Thanks for using Huawei Enterprise Support Community! We will help you learn how we collect, use, store and share your personal information and the rights you have in accordance with Privacy Policy and User Agreement.