Running a Spark SQL in Windows 7 connecting Yarn-Client - apache-spark-sql

Question

Am trying to execute a Spark SQL job in my Windows machine where the Spark is installed. Also am trying to connect to my Hadoop Cluster through yarn-client. Somehow my job is throwing below error. Am not sure why this is not connecting to my hortonworks sandbox.
I copied core-site.xml, yarn-site.xml, hive-site.xml into my conf folder in Windows. And I have setup SPARK_CONF and HADOOP_CONF_DIR in system environment variables. Please advise.
Error:
15/09/04 14:06:14 INFO HttpServer: Starting HTTP Server
15/09/04 14:06:14 INFO Utils: Successfully started service 'HTTP file server' on port 56407.
15/09/04 14:06:14 INFO SparkEnv: Registering OutputCommitCoordinator
15/09/04 14:06:15 INFO Utils: Successfully started service 'SparkUI' on port 4040.
15/09/04 14:06:15 INFO SparkUI: Started SparkUI at http://169.254.196.102:4040
15/09/04 14:06:18 INFO RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
15/09/04 14:06:20 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
15/09/04 14:06:22 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
15/09/04 14:06:24 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
15/09/04 14:06:26 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
15/09/04 14:06:28 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
Scala Code:
import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.{SparkConf, SparkContext}
/**
* Created by gfp2ram on 9/3/2015.
*/
object HiveConnect {
def main(args: Array[String]) {
System.setProperty("hadoop.home.dir", "C:\\hadoop\\hadoop-common-2.2.0-bin-master\\")
// val conf = new SparkConf().setAppName("HiveConnect").setMaster("yarn-client").set("spark.yarn.access.namenodes","hdfs://127.0.0.1:8020")
val conf = new SparkConf().setAppName("HiveConnect").setMaster("yarn-client")
val sc = new SparkContext(conf)
val sqlContext = new HiveContext(sc)
val df = sqlContext.sql("create table yahoo_orc_table (date STRING, open_price FLOAT, high_price FLOAT, low_price FLOAT, close_price FLOAT, volume INT, adj_price FLOAT) stored as orc")
val yahoo_stocks = sc.textFile("hdfs://127.0.0.1:8020/tmp/yahoo_stocks.csv")
val header = yahoo_stocks.first
val data = yahoo_stocks.filter(_(0) != header(0))
case class YahooStockPrice(date: String, open: Float, high: Float, low: Float, close: Float, volume: Integer, adjClose: Float)
val stockprice = data.map(_.split(",")).map(row => YahooStockPrice(row(0), row(1).trim.toFloat, row(2).trim.toFloat, row(3).trim.toFloat, row(4).trim.toFloat, row(5).trim.toInt, row(6).trim.toFloat))
println(stockprice)
stockprice.registerTempTable("yahoo_stocks_temp")
val results = sqlContext.sql("SELECT * FROM yahoo_stocks_temp")
results.map(t => "Stock Entry: " + t.toString).collect().foreach(println)
results.saveAsOrcFile("yahoo_stocks_orc")
val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
val yahoo_stocks_orc = sqlContext.orcFile("yahoo_stocks_orc")
yahoo_stocks_orc.registerTempTable("orcTest")
sqlContext.sql("SELECT * from orcTest").collect.foreach(println)
}
}

Answer 1

I am not able to add a comment due to lack of reputation points. However, Looking from the logs it's seems that spark is not able to connect to resource manager.
15/09/04 14:06:15 INFO SparkUI: Started SparkUI at http://169.254.196.102:4040
15/09/04 14:06:18 INFO RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
Can you ping to and check to make sure the resourcemanager is accessible from the windows mahine.
In all likelhood that would be the cause of the issue.