Spark HiveContext - reading from external partitioned Hive table delimiter issue - hive

Question

I have an external partitioned Hive table with underling file ROW FORMAT DELIMITED FIELDS TERMINATED BY '|'
Reading data via Hive directly is just fine, but when using Spark's Dataframe API the delimiter '|' is not taken into consideration.
Create external partitioned table:
hive> create external table external_delimited_table(value1 string, value2 string)
partitioned by (year string, month string, day string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '|'
location '/client/edb/poc_database/external_delimited_table';
Create data file containg just one row and place it to external partitioned table location:
shell>echo "one|two" >> table_data.csv
shell>hadoop fs -mkdir -p /client/edb/poc_database/external_delimited_table/year=2016/month=08/day=20
shell>hadoop fs -copyFromLocal table_data.csv /client/edb/poc_database/external_delimited_table/year=2016/month=08/day=20
Make partition active:
hive> alter table external_delimited_table add partition (year='2016',month='08',day='20');
Sanity check:
hive> select * from external_delimited_table;
select * from external_delimited_table;
+----------------------------------+----------------------------------+--------------------------------+---------------------------------+-------------------------------+--+
| external_delimited_table.value1 | external_delimited_table.value2 | external_delimited_table.year | external_delimited_table.month | external_delimited_table.day |
+----------------------------------+----------------------------------+--------------------------------+---------------------------------+-------------------------------+--+
| one | two | 2016 | 08 | 20
Spark code:
import org.apache.spark.sql.DataFrame
import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.{SparkContext, SparkConf}
object TestHiveContext {
def main(args: Array[String]): Unit = {
val conf = new SparkConf().setAppName("Test Hive Context")
val spark = new SparkContext(conf)
val hiveContext = new HiveContext(spark)
val dataFrame: DataFrame = hiveContext.sql("SELECT * FROM external_delimited_table")
dataFrame.show()
spark.stop()
}
dataFrame.show() output:
+-------+------+----+-----+---+
| value1|value2|year|month|day|
+-------+------+----+-----+---+
|one|two| null|2016| 08| 20|
+-------+------+----+-----+---+

Answer 1

This turned out to be a problem with Spark version 1.5.0. In version 1.6.0 issue doesn't take place:
scala> sqlContext.sql("select * from external_delimited_table")
res2: org.apache.spark.sql.DataFrame = [value1: string, value2: string, year: string, month: string, day: string]
scala> res2.show
+------+------+----+-----+---+
|value1|value2|year|month|day|
+------+------+----+-----+---+
| one| two|2016| 08| 20|
+------+------+----+-----+---+