Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error after submitting job to standalone cluster executor #139

Open
xiongtx opened this issue Dec 16, 2018 · 0 comments
Open

Error after submitting job to standalone cluster executor #139

xiongtx opened this issue Dec 16, 2018 · 0 comments

Comments

@xiongtx
Copy link

xiongtx commented Dec 16, 2018

I've a test project I'm trying to get working, but I can't figure out what's wrong.

After building an uberjar, I submit it to a local standalone Spark cluster:

spark-submit --master spark://txx.attlocal.net:7077 /<path to JAR>/sparktest-0.1.0-SNAPSHOT-standalone.jar

This results in an error on the executor:

org.apache.spark.SparkException: Failed to register classes with Kryo
	at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:140)
	at org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:324)
...
Caused by: java.lang.ClassNotFoundException: flambo.kryo.BaseFlamboRegistrator
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

However, flambo.kryo.BaseFlamboRegistrator is definitely in the JAR, and the job does execute properly when launched through a REPL:

spark-submit --master spark://txx.attlocal.net:7077 --class clojure.main /<path to JAR>/sparktest-0.1.0-SNAPSHOT-standalone.jar

user=> (require '[sparktest.core])
...

user=> (in-ns 'sparktest.core)
#object[clojure.lang.Namespace 0x11d2dd2d "sparktest.core"]

user=> (-main)
...
100
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant