spark-sql on google dataproc from sbt scala - apache-spark-sql

Question

Using Google Dataproc Spark cluster, my sbt built assembly jar can access Cassandra via SparkContext.
However, when I try to access via sqlContext I get spark sql classes not found on the remote cluster - though I believe the dataproc cluster is supposed to be provisioned for spark sql.
java.lang.NoClassDefFoundError: org/apache/spark/sql/types/UTF8String$
at org.apache.spark.sql.cassandra.CassandraSQLRow$$anonfun$fromJavaDriverRow$1.apply$mcVI$sp(CassandraSQLRow.scala:50)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala
my sbt file:
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.5.0" % "provided",
"org.apache.spark" %% "spark-sql" % "1.5.0" % "provided",
"com.datastax.spark" %% "spark-cassandra-connector" % "1.4.0"
)
Turning off "provided" on spark-sql puts me in jar duplicate merge hell.
Thx for any help.

Answer 1

It looks like you also need version 1.5.0 of the spark-cassandra-connector to ensure your classes are compatible. Here's the commit which upgraded the cassandra connector to 1.5.0, and you can see it removes the import of org.apache.spark.sql.types.UTF8String and adds import org.apache.spark.unsafe.types.UTF8String instead, changing the relevant lines in CassandraSQLRow.scala:
data(i) = GettableData.get(row, i)
data(i) match {
case date: Date => data.update(i, new Timestamp(date.getTime))
- case str: String => data.update(i, UTF8String(str))
+ case bigInt: BigInteger => data.update(i, new JBigDecimal(bigInt))
+ case str: String => data.update(i, UTF8String.fromString(str))
case set: Set[_] => data.update(i, set.toSeq)
case _ =>
}
Though it appears there are only "milestone" artifact types rather than "release" types in Maven central for the cassandra connector, you should still be able to get the latest milestone connector 1.5.0-M2 to work with your code.
EDIT: Additional link to the compatibility table from the Cassandra connector's GitHub README.md