how to pass in array into udf spark - sql

Question

I have a problem that 1) I don't really know how to call a registered UDF. I found some answer saying use callUDF so this is how I call the function in my code. 2) I don't really know how to pass in arrays as parameters.
Here is my code:
val df = Seq(("1","2","3","4","5","6")).toDF("A","B","C","D","E","F")
val newdf = Seq(("1","2","3","4","5","6")).toDF("A","B","C","D","E","F")
val cols = df.columns
val temp = Array(df.select($"A"),df.select($"B"),df.select($"C"),df.select($"D"),df.select($"E"),df.select($"F"))
val temp2 = Array(newdf.select($"A"),newdf.select($"B"),newdf.select($"C"),newdf.select($"D"),newdf.select($"E"),newdf.select($"F"))
sparkSession.udf.register ( "myfunc" , ((A:Array[String],B:Array[String]) => {for(i <- 0 to 5)yield( if (A(i)==B(i)) "U" else "N")} ) )
val a = df.withColumn("A",callUDF("myfunc",(temp,temp2)))
Thanks in advance!

Answer 1

You are trying to use columns from two different dataframes which is illegal in a UDF. Spark's UDF can only work on a per row basis. You can't combine rows from different dataframes. To do so you need to perform a join between the two.
In your case you have just one row but in a realistic case you would have multiple rows, you need to make sure you have some unique key to join by such as a unique id.
If you don't and both dataframes have the same number of rows and the same number of partitions you can easily create an id for both dataframes like this:
df.withColumn("id",monotonicallyIncreasingId)
You should probably also rename the columns to have different names.
Look at the different options for join (see http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.Dataset) to see what best matches your need.
As for registering and calling a udf you can do:
def myFunc(s1: Seq[Int], s2: Seq[Int]) = {
for(i <- 0 to 5) yield {
if (s1(i)==s2(i)) "U" else "N"
}
}
val u = udf(myFunc)
val a = df.withColumn("A", myFunc(temp,temp2))
note that temp and temp2 should each be a column representing an array in the same dataframe, i.e. you should define them after the join on the relevant columns.