How can I connect to hive using pyspark? - hive

Question

I'm trying to create a table in HIVE. But it is creating a folder like testdb.db inside spark-warehouse folder. How can I directly store in HIVE as we store to MySQL/MongoDB databases.
conf = SparkConf().setAppName("data_import")
sc = SparkContext(conf = conf)
sqlContext = HiveContext(sc)
sqlContext.setConf("spark.sql.shuffle.partitions", "2")
sqlContext.sql("CREATE DATABASE testdb")
sqlContext.sql("use testdb")
sqlContext.sql("create table daily_revenue(order_date string, daily_revenue float)")

Answer 1

When you creates a table in HIVE then what happens behind the scene is, it stores the metadata in some relational database depending on which is configured for your environment and actual data will be stored on HDFS warehouse directory if that is managed table.
Similarly when you try to create the table from Spark in HIVE then what it will do is, first it will create the folder .db and inside this folder it will create another folder with table name, which inturn store the data on HDFS.
So in your case, you should have <warehouse_dir>/testdb.db/table folder. and
if you load any data to this table, that will be present inside the table folder.
Hope it helps.
Regards,
Neeraj

Answer 2

sqlContext.sql("create database if not exists demo")
>>> sqlContext.sql("show tables in demo").show()
+---------+-----------+
|tableName|isTemporary|
+---------+-----------+
+---------+-----------+
sqlContext.sql("create table demo.dummy (id int, name string)")
>>> sqlContext.sql("show tables in demo").show()
+---------+-----------+
|tableName|isTemporary|
+---------+-----------+
| dummy| false|
+---------+-----------+
>>> sqlContext.sql("desc demo.dummy").show()
+--------+---------+-------+
|col_name|data_type|comment|
+--------+---------+-------+
| id| int| null|
| name| string| null|
+--------+---------+-------+