Spark: Multiple filter inside agg and concat not null values - apache-spark-sql

Question

I'm trying to concatenate not null values from a List column.
I know this can be done easily by using UDF but would like to know how to handle this by using multiple filter conditions inside agg function.
Don't know what's missing here?
val df = sc.parallelize(Seq(("foo", List(null,"bar",null)),
("bar", List("one","two",null)),
("rio", List("Ria","","Kevin")))).toDF("key", "value")
+---+-----------------+
|key| value|
+---+-----------------+
|foo|[null, bar, null]|
|bar| [one, two, null]|
|rio| [Ria, , Kevin]|
+---+-----------------+
df.groupBy("key")
.agg(concat_ws(",",first(when(($"value".isNotNull || $"value" =!= ""),$"value"))).as("RemovedNullSeq"))
.show(false)
+---+--------------+
|key|RemovedNullSeq|
+---+--------------+
|bar|one,two |
|rio|Ria,,Kevin |
|foo|bar |
+---+--------------+
I don't need that blank value in the second record.
Thanks

Answer 1

I'm not immediately sure if using aggregate functions is necessary based on the example provided.
If you're just trying to concatenate the values in an array then the following works:
val df = Seq(List(null,"abc", null),
List(null, null, null),
List(null, "def", "ghi", "kjl"),
List("mno", null, "pqr")).toDF("list")
df.withColumn("concat", concat_ws(",",$"list")).show(false)
+---------------------+-----------+
|list |concat |
+---------------------+-----------+
|[null, abc, null] |abc |
|[null, null, null] | |
|[null, def, ghi, kjl]|def,ghi,kjl|
|[mno, null, pqr] |mno,pqr |
+---------------------+-----------+
If there is a need to group first:
val df2 = Seq((123,List(null,"abc", null)),
(123,List(null,"def", "hij"))).toDF("key","list")
df2.show(false)
+---+-----------------+
|key|list |
+---+-----------------+
|123|[null, abc, null]|
|123|[null, def, hij] |
+---+-----------------+
You might think you could do something like
val grouped = df2.groupBy($"key").agg(collect_list($"list").as("collected"))
And then apply some functions to the array of arrays to obtain your concatenated result. However, I have been unable to find a way to do this without resorting to UDFs.
In this case, exploding before the grouping does the trick:
val grouped = df2.groupBy($"key").agg(collect_list($"list").as("collected"))
.groupBy($"key").agg(collect_list($"listItem").as("collected"))
.withColumn("concat", concat_ws(",",$"collected")).show(false)
+---+---------------+-----------+
|key|collected |concat |
+---+---------------+-----------+
|123|[abc, def, hij]|abc,def,hij|
+---+---------------+-----------+
Note however that there is no guarantee of the order in which the lists will be collected.
Hope this helps

Answer 2

import org.apache.spark.sql.functions._
val df = sc.parallelize(Seq(("foo", List(null,"bar",null)),
("bar", List("one","two",null)),
("rio", List("Ria","","Kevin")))).toDF("key", "value")
val filtd = df.select($"key" as "key", explode($"value") as "val").where (length($"val") > 0)
val rsult = filtd.select($"*").groupBy($"key").agg(collect_list("val"))
rsult.show(5)
You can add ultiple conditions like this
val filtd = df.select($"key" as "key", explode($"value") as "val").where (length($"val") > 0 && $"val".isNotNull)
Output
+---+-----------------+
|key|collect_list(val)|
+---+-----------------+
|bar| [one, two]|
|rio| [Ria, Kevin]|
|foo| [bar]|
+---+-----------------+