How run spark app with IntelliJ IDEA on spark cluster in mesos - intellij-idea

Question

I have a spark cluster on mesos with 1 master and 1 slave on the same host.
Then I can execute official spark examples with spark-submit as follow:
//bin/spark-submit --deploy-mode cluster --master mesos://<master_ip>:7077 --class org.apache.spark.examples.SparkPi /opt/spark/lib/spark-examples-1.4.0-hadoop2.6.0.jar
Also I try to build app with IntelliJ IDEA. When I execute code on locall machine:
import org.apache.spark.{SparkConf, SparkContext}
object SimpleApp {
def main(args: Array[String]) {
val conf = new SparkConf()
.setAppName("Simple Application")
.setMaster("local")
val sc = new SparkContext(conf)
...
}
}
All run fine locally, but when change to run on spark-mesos:
import org.apache.spark.{SparkConf, SparkContext}
object SimpleApp {
def main(args: Array[String]) {
val conf = new SparkConf()
.setAppName("Simple Application")
.setMaster("mesos://<master_ip>:7077")
val sc = new SparkContext(conf)
...
}
}
The output error it's:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/myusername/project/spark-01/lib/spark-assembly-1.4.0-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/myusername/project/spark-01/lib/spark-examples-1.4.0-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/08/18 16:29:53 INFO SparkContext: Running Spark version 1.4.0
15/08/18 16:29:54 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/08/18 16:29:54 INFO SecurityManager: Changing view acls to: myusername
15/08/18 16:29:54 INFO SecurityManager: Changing modify acls to: myusername
15/08/18 16:29:54 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(myusername); users with modify permissions: Set(myusername)
15/08/18 16:29:56 INFO Slf4jLogger: Slf4jLogger started
15/08/18 16:29:56 INFO Remoting: Starting remoting
15/08/18 16:29:56 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver#172.23.10.21:37362]
15/08/18 16:29:56 INFO Utils: Successfully started service 'sparkDriver' on port 37362.
15/08/18 16:29:56 INFO SparkEnv: Registering MapOutputTracker
15/08/18 16:29:56 INFO SparkEnv: Registering BlockManagerMaster
15/08/18 16:29:57 INFO DiskBlockManager: Created local directory at /tmp/spark-29fef56b-0a26-4cd7-b391-2f436bca1c55/blockmgr-b7febe40-5d37-4862-be78-4b6f4df1738c
15/08/18 16:29:57 INFO MemoryStore: MemoryStore started with capacity 953.4 MB
15/08/18 16:29:57 INFO HttpFileServer: HTTP File server directory is /tmp/spark-29fef56b-0a26-4cd7-b391-2f436bca1c55/httpd-94618d51-782f-4262-a113-8d44bf0b29d7
15/08/18 16:29:57 INFO HttpServer: Starting HTTP Server
15/08/18 16:29:57 INFO Utils: Successfully started service 'HTTP file server' on port 59838.
15/08/18 16:29:57 INFO SparkEnv: Registering OutputCommitCoordinator
15/08/18 16:29:57 INFO Utils: Successfully started service 'SparkUI' on port 4040.
15/08/18 16:29:57 INFO SparkUI: Started SparkUI at http://172.23.10.21:4040
Failed to load native Mesos library from /home/myusername/current/idea.14/bin::/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
Exception in thread "main" java.lang.UnsatisfiedLinkError: no mesos in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1886)
at java.lang.Runtime.loadLibrary0(Runtime.java:849)
at java.lang.System.loadLibrary(System.java:1088)
at org.apache.mesos.MesosNativeLibrary.load(MesosNativeLibrary.java:54)
at org.apache.mesos.MesosNativeLibrary.load(MesosNativeLibrary.java:79)
at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2535)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:489)
at SimpleApp$.main(SimpleApp.scala:11)
at SimpleApp.main(SimpleApp.scala)
I solve the problem importing all .so dependencies of mesos, but not is a pretty solution and then al developers need to know mesos .so
I research about: How run spark app on IntelliJ IDEA but all examples show the first scenario runing locally.
Questions:
Is it the valid scenario to develop an sapark app? or the rigth flow is develop algorith locally and later use spark-submit to run on mesos?
Anybody know a better way to run spark app from IntelliJ IDEA to run on spark mesos cluster?