How to do GROUP BY on exploded field in Spark SQL's? - sql

Question

Zeppelin 0.6
Spark 1.6
SQL
I am trying to find the top 20 occurring words in some tweets. filtered contains an array of words for each tweet. The following:
select explode(filtered) AS words from tweettable
lists each word as you would expect, but what I want is to get a count of each word over all tweets and then display the top 20 of these. The following works but I need to do this in SQL:
df.select(explode($"filtered").as("value"))
.groupBy("value")
.count()
.sort(desc("count"))
.show(20, false)
I tried GROUP BY on words, filtered, and explode(filtered) but all gave errors.

Answer 1

You can use subqueries in the FROM statement:
SELECT value, count(*) AS count
FROM (SELECT explode(filtered) AS value
FROM tweettable) AS temp
GROUP BY value
ORDER BY count DESC

Answer 2

Following code will give you complete idea to achieve what you are expecting. Tested in Spark(1.6)
val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
import hiveContext.implicits._
val lst = List(Seq("Hello","Hyd","Hello","Mumbai"),Seq("Hello","Mumbai"),Seq("Hello","Delhi","Hello","Banglore"))
case class Tweets(filtered: Seq[String])
val df = sc.parallelize(lst).map(x=>Tweets(x)).toDF
import org.apache.spark.sql.functions.{explode}
import org.apache.spark.sql.functions.count
df.select(explode($"filtered").as("value")).groupBy("value").agg(count("*").alias("cnt")).orderBy('cnt.desc).show(20,false)
Alternatively you can use window function.
val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
import hiveContext.implicits._
val lst = List(Seq("Hello","Hyd","Hello","Mumbai"),Seq("Hello","Mumbai"),Seq("Hello","Delhi","Hello","Banglore"))
case class Tweets(filtered: Seq[String])
val df = sc.parallelize(lst).map(x=>Tweets(x)).toDF
import org.apache.spark.sql.expressions.Window
import org.apache.spark.sql.functions._
val w = org.apache.spark.sql.expressions.Window.orderBy('cnt.desc)
df.select(explode($"filtered").as("value")).groupBy("value").agg(count("*").alias("cnt")).withColumn("filteredrank", rank.over(w)).filter(col("filteredrank") <= 20).show()