Got it

Huawei Enterprise Support Community

Community Forums Cloud & Big Data

error code as following:

error code as following:

Created: Jun 30, 2019 15:28:05Latest reply: Jun 30, 2019 15:33:39 306 1 1 0 0

Rewarded HiCoins： 0 (problem resolved)

View the author ^1#

Exception in thread "main" org.apache.spark.SparkException: Task not serializable

at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:298)

at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:288)

at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:108)

at org.apache.spark.SparkContext.clean(SparkContext.scala:2094)

at org.apache.spark.rdd.RDD$$anonfun$map$1.apply(RDD.scala:370)

at org.apache.spark.rdd.RDD$$anonfun$map$1.apply(RDD.scala:369)

at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)

at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)

at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)

at org.apache.spark.rdd.RDD.map(RDD.scala:369)

at com.haizhi.test.ClassB.fun(SerializableTest.scala:18)

at com.haizhi.test.SerializableTest$.main(SerializableTest.scala:35)

at com.haizhi.test.SerializableTest.main(SerializableTest.scala)

Caused by: java.io.NotSerializableException: org.apache.spark.SparkContext

Serialization stack:

- object not serializable (class: org.apache.spark.SparkContext, value: org.apache.spark.SparkContext@433e536f)

- field (class: com.haizhi.test.ClassB, name: sc, type: class org.apache.spark.SparkContext)

- object (class com.haizhi.test.ClassB, com.haizhi.test.ClassB@667e34b1)

- field (class: com.haizhi.test.ClassB$$anonfun$fun$1, name: $outer, type: class com.haizhi.test.ClassB)

- object (class com.haizhi.test.ClassB$$anonfun$fun$1, <function1>)

at org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40)

at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:46)

at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:100)

at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:295)

... 12 more

x
convention：
Emotion(0)

Like (1) Dislike (0) Favorite(0) Report

 Previous: Caused by: java.lang.NullPointerException

Next: Flink error

Featured Answers

Best answer

(4) (0)

tanjingmeng

Created Jun 30, 2019 15:33:39

Hey there!

Method 1

Modify ClassA to inherit the Serializable class.

class ClassA extends Serializable {

def getClassName: String = this.getClass.getSimpleName

}

or try this one:

rdd.map(i => {

val classA = new ClassA

"getClassName in main -> " + classA.getClassName + s": $i"

}).collect().foreach(println)

View more

x
convention：

All Answers

(4) (0)

^2#

tanjingmeng

Created Jun 30, 2019 15:33:39

Hey there!

Method 1

Modify ClassA to inherit the Serializable class.

class ClassA extends Serializable {

def getClassName: String = this.getClass.getSimpleName

}

or try this one:

rdd.map(i => {

val classA = new ClassA

"getClassName in main -> " + classA.getClassName + s": $i"

}).collect().foreach(println)

View more

x
convention：

Comment

Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:

Politically sensitive content
Content concerning pornography, gambling, and drug abuse
Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy

Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " User Agreement."

My Followers

Are you sure to block this user?
Users on your blacklist cannot comment on your post,cannot mention you, cannot send you private messages.

Please bind your phone number to obtain invitation bonus.