Cannot cast dataframe column containing an array to String - dataframe

Question

I have the following dataframe:
I wouldl ike to transform the results column into another dataframe.
This is the code I am trying to execute:
val JsonString = df.select(col("results")).as[String]
val resultsDF = spark.read.json(JsonString)
But the first line returns this error:
AnalysisException: Cannot up cast `results` from array<struct<auctions:bigint,bid_price_sum:double,bid_selected_price_sum:double,bids_cancelled:bigint,bids_done:bigint,bids_fail_currency:bigint,bids_fail_parsing:bigint,bids_failed:bigint,bids_filtered_blockrule:bigint,bids_filtered_duration:bigint,bids_filtered_floor_price:bigint,bids_lost:bigint,bids_selected:bigint,bids_timeout:bigint,clicks:bigint,content_owner_id:string,content_owner_name:string,date:bigint,impressions:bigint,intext_inventory:bigint,ivt_blocked:struct<blocked_reason_automated_browsing:bigint,blocked_reason_data_center:bigint,blocked_reason_false_representation:bigint,blocked_reason_irregular_pattern:bigint,blocked_reason_known_crawler:bigint,blocked_reason_manipulated_behavior:bigint,blocked_reason_misleading_uer_interface:bigint,blocked_reason_undisclosed_classification:bigint,blocked_reason_undisclosed_classification_ml:bigint,blocked_reason_undisclosed_use_of_incentives:bigint,ivt_blocked_requests:bigint>,no_bid:bigint,requests:bigint,requests_country:bigint,revenue:double,vtr0:bigint,vtr100:bigint,vtr25:bigint,vtr50:bigint,vtr75:bigint>> to string.
The type path of the target object is:
- root class: "java.lang.String"
You can either add an explicit cast to the input data or choose a higher precision type of the field in the target object

Answer 1

This means that results is not a String.
For example
import org.apache.spark.sql.{DataFrame, SparkSession}
import org.apache.spark.sql.functions.col
object Main extends App {
val spark = SparkSession.builder
.master("local")
.appName("Spark app")
.getOrCreate()
import spark.implicits._
case class MyClass(auctions: Int, bid_price_sum: Double)
val df: DataFrame =
Seq(
("xxx", "yyy", 1, """[{"auctions":9343, "bid_price_sum":1.062}, {"auctions":1225, "bid_price_sum":0.153}]"""),
("xxx1", "yyy1", 2, """{"auctions":1111, "bid_price_sum":0.111}"""),
)
.toDF("col1", "col2", "col3", "results")
df.show()
val JsonString = df.select(col("results")).as[String]
val resultsDF = spark.read.json(JsonString)
resultsDF.show()
}
produces
+----+----+----+--------------------+
|col1|col2|col3| results|
+----+----+----+--------------------+
| xxx| yyy| 1|[{"auctions":9343...|
|xxx1|yyy1| 2|{"auctions":1111,...|
+----+----+----+--------------------+
+--------+-------------+
|auctions|bid_price_sum|
+--------+-------------+
| 9343| 1.062|
| 1225| 0.153|
| 1111| 0.111|
+--------+-------------+
while
// ........................
val df: DataFrame =
Seq(
("xxx", "yyy", 1, Seq(MyClass(9343, 1.062), MyClass(1225, 0.153))),
("xxx1", "yyy1", 2, Seq(MyClass(1111, 0.111))),
)
.toDF("col1", "col2", "col3", "results")
// ........................
produces your exception
org.apache.spark.sql.AnalysisException: Cannot up cast results
from "ARRAY<STRUCT<auctions: INT, bid_price_sum: DOUBLE>>" to "STRING".
The type path of the target object is:
- root class: "java.lang.String"
You can either add an explicit cast to the input data
or choose a higher precision type of the field in the target object
You can fix the exception with
df.select(col("results")).as[Seq[MyClass]]
instead of
df.select(col("results")).as[String]
Similarly,
val df = spark.read.json("src/main/resources/file.json")
in the above code produces correct result for the following file.json
{"col1": "xxx", "col2": "yyy", "col3": 1, "results": "[{\"auctions\":9343, \"bid_price_sum\":1.062}, {\"auctions\":1225, \"bid_price_sum\":0.153}]"}
{"col1": "xxx1", "col2": "yyy1", "col3": 2, "results": "{\"auctions\":1111, \"bid_price_sum\":0.111}"}
but throws for the following file
{"col1": "xxx", "col2": "yyy", "col3": 1, "results": [{"auctions":9343, "bid_price_sum":1.062}, {"auctions":1225, "bid_price_sum":0.153}]}
{"col1": "xxx1", "col2": "yyy1", "col3": 2, "results": [{"auctions":1111, "bid_price_sum":0.111}]}
Extract columns from a json and write into a dataframe