Aggregate by struct field in spark scala - dataframe

I have dataframe like this:
+-------+--------------+-----------------------+
|column1|mapColumn |arrayColumn |
+-------+--------------+-----------------------+
|a |[b -> 100.1] |[d] |
|a |[c -> 200.1] |[d, e] |
+-------+--------------+-----------------------+
val groupFunc = udf((x: Seq[Map[String, Double]]) => x.flatten.toMap)
df.groupBy("column1")
.agg(
map(col("column1"),
struct(
groupFunc(collect_list("mapColumn")),
flatten(collect_set(col("arrayColumn")))
)
).as("aggregatedMap")
)
After the aggregation the result is follow:
+-------+--------------------------------------------+
|column1|aggregatedMap |
+-------+--------------------------------------------+
|a |[a -> [[b -> 100.1, c -> 200.1], [d, d, e]]]|
+-------+--------------------------------------------+
Expected result:
+-------+--------------------------------------------+
|column1|aggregatedMap |
+-------+--------------------------------------------+
|a |[a -> [[b -> 100.1, c -> 200.1], [d, e]]] |
+-------+--------------------------------------------+
Any ideas how to remove duplicates while aggregating?

Related

Transform Map[String,Any] to a dataframe in Scala

I have the following object of type Map[String,Any]:
d: Map[String,Any] = Map(count -> 1, results -> List(Map(A -> 1, C -> Hello, B -> Map(BA -> 0, BC -> 0)), Map(A -> 2, C -> Hi, B -> Map(BA -> 0, BC -> 0))), parameters -> Map(P1 -> 805, P2 -> 20230101))
I dont need most of these informations. I only need to extract results map list into a dataframe (ignoring the B object as it is a map). So the desired output would be:
+---------+----------------+
| A |C |
+---------+----------------+
| 1 |Hello |
| 2 |Hi |
+---------+----------------+
I tried:
val df = d
.map( m => (m.get("A"),m.get("C")))
.toDF("A", "C")
But I got
error: value get is not a member of Any

Try casting .asInstanceOf[Seq[Map[String, Any]]], .asInstanceOf[(Int, String)]
import org.apache.spark.sql.SparkSession
val d: Map[String,Any] = Map(
"count" -> 1,
"results" -> List(
Map("A" -> 1, "C" -> "Hello", "B" -> Map("BA" -> 0, "BC" -> 0)),
Map("A" -> 2, "C" -> "Hi", "B" -> Map("BA" -> 0, "BC" -> 0))
),
"parameters" -> Map("P1" -> 805, "P2" -> 20230101)
)
val spark = SparkSession.builder
.master("local")
.appName("Spark app")
.getOrCreate()
import spark.implicits._
val df = d("results")
.asInstanceOf[Seq[Map[String, Any]]]
.map(m =>
(m("A"), m("C")).asInstanceOf[(Int, String)]
)
.toDF("A", "C")
df.show()
//+---+-----+
//| A| C|
//+---+-----+
//| 1|Hello|
//| 2| Hi|
//+---+-----+

Transform list of map to dataframe

I have the following data:
d = Some(List(Map(id -> 1, n -> Hi), Map(id -> 2, n -> Hello)))
I would like to transform it into a dataframe like the following:
+--------+
|id|n |
+--------+
|1 |Hi |
+--------+
|2 |Hello|
+--------+
I tried the following:
import spark.implicits._
val df = d
.map( m => (m.get("id"),m.get("n")))
.toDF("id", "n")
But im getting:
error: value get is not a member of Any
.map( m => (m.get("id"),m.get("n")))

Your top level here is Option and i think thats the reason why you cant handle it with single map. I managed to do it with something like this:
import spark.implicits._
val d = Some(List(Map("id" -> "1", "n" -> "Hi"), Map("id" -> "2", "n" -> "Hello")))
val data = d.fold (List.empty [(Option [String], Option [String])]) (_.map (m => (m.get ("id"), m.get ("n"))))
val df = data.toDF("id", "n").show()
Output:
+---+-----+
| id| n|
+---+-----+
| 1| Hi|
| 2|Hello|
+---+-----+

Spark: How to filter out data based on subset condition

I have two table, p_to_v mapping, g_to_v mapping.
scala> val p_to_v = Seq(("p1", "v1"), ("p1", "v2"), ("p2", "v1")).toDF("p", "v")
scala> p_to_v.show
+---+---+
| p| v|
+---+---+
| p1| v1|
| p1| v2|
| p2| v1|
+---+---+
"p1" is mapped to [v1, v2]
"p2" is mapped to [v1]
scala> val g_to_v = Seq(("g1", "v1"), ("g2", "v1"), ("g2", "v2"), ("g3", "v2")).toDF("g", "v")
scala> g_to_v.show
+---+---+
| g| v|
+---+---+
| g1| v1|
| g2| v1|
| g2| v2|
| g3| v2|
+---+---+
"g1" is mapped to [v1]
"g2" is mapped to [v1,v2]
"g3" is mapped to [v2]
I want to get all p and g combination for which corresponding v mapping of p is subset of v mapping of g
p1 [v1, v2] - g2 [v1, v2]
p2 [v1] - g1 [v1]
p2 [v1] - g2 [v1, v2]
How can I get same?

This is pretty straightforward. You need to use groupBy & then simple inner join
scala> val p_to_v = Seq(("p1", "v1"), ("p1", "v2"), ("p2", "v1")).toDF("p", "v")
19/10/16 22:11:55 WARN metastore: Failed to connect to the MetaStore Server...
p_to_v: org.apache.spark.sql.DataFrame = [p: string, v: string]
scala> val g_to_v = Seq(("g1", "v1"), ("g2", "v1"), ("g2", "v2"), ("g3", "v2")).toDF("g", "v")
g_to_v: org.apache.spark.sql.DataFrame = [g: string, v: string]
Now do groupBy operation
scala> val pv = p_to_v.groupBy($"p").agg(collect_list("v").as("pv"))
pv: org.apache.spark.sql.DataFrame = pv = [p: string, pv: array<string>]
scala> val gv = g_to_v.groupBy($"g").agg(collect_list("v").as("gv"))
gv: org.apache.spark.sql.DataFrame = [g: string, gv: array<string>]
scala> pv.show
+---+--------+
| p| pv|
+---+--------+
| p2| [v1]|
| p1|[v1, v2]|
+---+--------+
scala> gv.show
+---+--------+
| g| gv|
+---+--------+
| g2|[v2, v1]|
| g3| [v2]|
| g1| [v1]|
+---+--------+
Create an UDF for finding the subset
import org.apache.spark.sql.Row
import org.apache.spark.sql.functions._
def subLis(ar1: Seq[Row], ar2: Seq[Row]) = ar1.toSet.subsetOf(ar2.toSet)
subLis: (ar1: Seq[org.apache.spark.sql.Row], ar2: Seq[org.apache.spark.sql.Row])Boolean
val subLisUDF = udf(subLis _)
UserDefinedFunction(<function2>,BooleanType,None)
Now you can perform a cross join & apply the UDF
spark.conf.set("spark.sql.crossJoin.enabled", "true")
pv.join(gv).withColumn("newdsa", subLisUDF($"pv", $"gv")).filter($"newdsa").show
+---+--------+---+--------+------+
| p| pv| g| gv|newdsa|
+---+--------+---+--------+------+
| p2| [v1]| g2|[v2, v1]| true|
| p1|[v1, v2]| g2|[v2, v1]| true|
| p2| [v1]| g1| [v1]| true|
+---+--------+---+--------+------+
Or joins with condition
pv.join(gv, pv("pv") === gv("gv") || subLisUDF($"pv", $"gv")).show
+---+--------+---+--------+
| p| pv| g| gv|
+---+--------+---+--------+
| p2| [v1]| g2|[v1, v2]|
| p1|[v1, v2]| g2|[v1, v2]|
| p2| [v1]| g1| [v1]|
+---+--------+---+--------+
Try both and take the best performing one.

Spark Advanced Window with dynamic last

Problem:
Given a time series data which is a clickstream of user activity is stored in hive, ask is to enrich the data with session id using spark.
Session Definition
Session expires after inactivity of 1 hour
Session remains active for a total duration of 2 hours
Data:
click_time,user_id
2018-01-01 11:00:00,u1
2018-01-01 12:10:00,u1
2018-01-01 13:00:00,u1
2018-01-01 13:50:00,u1
2018-01-01 14:40:00,u1
2018-01-01 15:30:00,u1
2018-01-01 16:20:00,u1
2018-01-01 16:50:00,u1
2018-01-01 11:00:00,u2
2018-01-02 11:00:00,u2
Below is partial solution considering only 1st point in session definition:
val win1 = Window.partitionBy("user_id").orderBy("click_time")
val sessionnew = when((unix_timestamp($"click_time") - unix_timestamp(lag($"click_time",1,"2017-01-01 11:00:00.0").over(win1)))/60 >= 60, 1).otherwise(0)
userActivity
.withColumn("session_num",sum(sessionnew).over(win1))
.withColumn("session_id",concat($"user_id", $"session_num"))
.show(truncate = false)
Actual Output:
+---------------------+-------+-----------+----------+
|click_time |user_id|session_num|session_id|
+---------------------+-------+-----------+----------+
|2018-01-01 11:00:00.0|u1 |1 |u11 |
|2018-01-01 12:10:00.0|u1 |2 |u12 | -- session u12 starts
|2018-01-01 13:00:00.0|u1 |2 |u12 |
|2018-01-01 13:50:00.0|u1 |2 |u12 |
|2018-01-01 14:40:00.0|u1 |2 |u12 | -- this should be a new session as diff of session start of u12 and this row exceeds 2 hours
|2018-01-01 15:30:00.0|u1 |2 |u12 |
|2018-01-01 16:20:00.0|u1 |2 |u12 |
|2018-01-01 16:50:00.0|u1 |2 |u12 | -- now this has to be compared with row 5 to find difference
|2018-01-01 11:00:00.0|u2 |1 |u21 |
|2018-01-02 11:00:00.0|u2 |2 |u22 |
+---------------------+-------+-----------+----------+
To include the second condition, I tried to find difference between the current time with last session start time to check if that exceeds 2 hours, but however the reference itself changes for the following rows. These are some some use cases which can be achieved through running sum but this doesn’t suit here.

Not a straight forward problem to solve, but here's one approach:
Use Window lag timestamp difference to identify sessions (with 0 = start of a session) per user for rule #1
Group the dataset to assemble the timestamp diff list per user
Process via a UDF the timestamp diff list to identify sessions for rule #2 and create all session ids per user
Expand the grouped dataset via Spark's explode
Sample code below:
import org.apache.spark.sql.functions._
import org.apache.spark.sql.expressions.Window
import spark.implicits._
val userActivity = Seq(
("2018-01-01 11:00:00", "u1"),
("2018-01-01 12:10:00", "u1"),
("2018-01-01 13:00:00", "u1"),
("2018-01-01 13:50:00", "u1"),
("2018-01-01 14:40:00", "u1"),
("2018-01-01 15:30:00", "u1"),
("2018-01-01 16:20:00", "u1"),
("2018-01-01 16:50:00", "u1"),
("2018-01-01 11:00:00", "u2"),
("2018-01-02 11:00:00", "u2")
).toDF("click_time", "user_id")
def clickSessList(tmo: Long) = udf{ (uid: String, clickList: Seq[String], tsList: Seq[Long]) =>
def sid(n: Long) = s"$uid-$n"
val sessList = tsList.foldLeft( (List[String](), 0L, 0L) ){ case ((ls, j, k), i) =>
if (i == 0 || j + i >= tmo) (sid(k + 1) :: ls, 0L, k + 1) else
(sid(k) :: ls, j + i, k)
}._1.reverse
clickList zip sessList
}
Note that the accumulator for foldLeft in the UDF is a Tuple of (ls, j, k), where:
ls is the list of formatted session ids to be returned
j and k are for carrying over the conditionally changing timestamp value and session id number, respectively, to the next iteration
Step 1:
val tmo1: Long = 60 * 60
val tmo2: Long = 2 * 60 * 60
val win1 = Window.partitionBy("user_id").orderBy("click_time")
val df1 = userActivity.
withColumn("ts_diff", unix_timestamp($"click_time") - unix_timestamp(
lag($"click_time", 1).over(win1))
).
withColumn("ts_diff", when(row_number.over(win1) === 1 || $"ts_diff" >= tmo1, 0L).
otherwise($"ts_diff")
)
df1.show
// +-------------------+-------+-------+
// | click_time|user_id|ts_diff|
// +-------------------+-------+-------+
// |2018-01-01 11:00:00| u1| 0|
// |2018-01-01 12:10:00| u1| 0|
// |2018-01-01 13:00:00| u1| 3000|
// |2018-01-01 13:50:00| u1| 3000|
// |2018-01-01 14:40:00| u1| 3000|
// |2018-01-01 15:30:00| u1| 3000|
// |2018-01-01 16:20:00| u1| 3000|
// |2018-01-01 16:50:00| u1| 1800|
// |2018-01-01 11:00:00| u2| 0|
// |2018-01-02 11:00:00| u2| 0|
// +-------------------+-------+-------+
Steps 2-4:
val df2 = df1.
groupBy("user_id").agg(
collect_list($"click_time").as("click_list"), collect_list($"ts_diff").as("ts_list")
).
withColumn("click_sess_id",
explode(clickSessList(tmo2)($"user_id", $"click_list", $"ts_list"))
).
select($"user_id", $"click_sess_id._1".as("click_time"), $"click_sess_id._2".as("sess_id"))
df2.show
// +-------+-------------------+-------+
// |user_id|click_time |sess_id|
// +-------+-------------------+-------+
// |u1 |2018-01-01 11:00:00|u1-1 |
// |u1 |2018-01-01 12:10:00|u1-2 |
// |u1 |2018-01-01 13:00:00|u1-2 |
// |u1 |2018-01-01 13:50:00|u1-2 |
// |u1 |2018-01-01 14:40:00|u1-3 |
// |u1 |2018-01-01 15:30:00|u1-3 |
// |u1 |2018-01-01 16:20:00|u1-3 |
// |u1 |2018-01-01 16:50:00|u1-4 |
// |u2 |2018-01-01 11:00:00|u2-1 |
// |u2 |2018-01-02 11:00:00|u2-2 |
// +-------+-------------------+-------+
Also note that click_time is "passed thru" in steps 2-4 so as to be included in the final dataset.

Though the answer provided by Leo Works perfectly I feel its a complicated approach to Solve the problem by using Collect and Explode functions.This can be solved using Spark's Way by using UDAF to make it feasible for modifications in the near future as well.Please take a look into a solution on similar lines below
scala> //Importing Packages
scala> import org.apache.spark.sql.expressions.Window
import org.apache.spark.sql.expressions.Window
scala> import org.apache.spark.sql.functions._
import org.apache.spark.sql.functions._
scala> import org.apache.spark.sql.types._
import org.apache.spark.sql.types._
scala> // CREATE UDAF To Calculate total session duration Based on SessionIncativeFlag and Current Session Duration
scala> import org.apache.spark.sql.expressions.MutableAggregationBuffer
import org.apache.spark.sql.expressions.MutableAggregationBuffer
scala> import org.apache.spark.sql.expressions.UserDefinedAggregateFunction
import org.apache.spark.sql.expressions.UserDefinedAggregateFunction
scala> import org.apache.spark.sql.Row
import org.apache.spark.sql.Row
scala> import org.apache.spark.sql.types._
import org.apache.spark.sql.types._
scala>
scala> class TotalSessionDuration extends UserDefinedAggregateFunction {
| // This is the input fields for your aggregate function.
| override def inputSchema: org.apache.spark.sql.types.StructType =
| StructType(
| StructField("sessiondur", LongType) :: StructField(
| "inactivityInd",
| IntegerType
| ) :: Nil
| )
|
| // This is the internal fields you keep for computing your aggregate.
| override def bufferSchema: StructType = StructType(
| StructField("sessionSum", LongType) :: Nil
| )
|
| // This is the output type of your aggregatation function.
| override def dataType: DataType = LongType
|
| override def deterministic: Boolean = true
|
| // This is the initial value for your buffer schema.
| override def initialize(buffer: MutableAggregationBuffer): Unit = {
| buffer(0) = 0L
| }
|
| // This is how to update your buffer schema given an input.
| override def update(buffer: MutableAggregationBuffer, input: Row): Unit = {
| if (input.getAs[Int](1) == 1)
| buffer(0) = 0L
| else if (buffer.getAs[Long](0) >= 7200L)
| buffer(0) = input.getAs[Long](0)
| else
| buffer(0) = buffer.getAs[Long](0) + input.getAs[Long](0)
| }
|
| // This is how to merge two objects with the bufferSchema type.
| override def merge(buffer1: MutableAggregationBuffer, buffer2: Row): Unit = {
| if (buffer2.getAs[Int](1) == 1)
| buffer1(0) = 0L
| else if (buffer2.getAs[Long](0) >= 7200)
| buffer1(0) = buffer2.getAs[Long](0)
| else
| buffer1(0) = buffer1.getAs[Long](0) + buffer2.getAs[Long](0)
| }
| // This is where you output the final value, given the final value of your bufferSchema.
| override def evaluate(buffer: Row): Any = {
| buffer.getLong(0)
| }
| }
defined class TotalSessionDuration
scala> //Create handle for using the UDAD Defined above
scala> val sessionSum=spark.udf.register("sessionSum", new TotalSessionDuration)
sessionSum: org.apache.spark.sql.expressions.UserDefinedAggregateFunction = TotalSessionDuration#64a9719a
scala> //Create Session Dataframe
scala> val clickstream = Seq(
| ("2018-01-01T11:00:00Z", "u1"),
| ("2018-01-01T12:10:00Z", "u1"),
| ("2018-01-01T13:00:00Z", "u1"),
| ("2018-01-01T13:50:00Z", "u1"),
| ("2018-01-01T14:40:00Z", "u1"),
| ("2018-01-01T15:30:00Z", "u1"),
| ("2018-01-01T16:20:00Z", "u1"),
| ("2018-01-01T16:50:00Z", "u1"),
| ("2018-01-01T11:00:00Z", "u2"),
| ("2018-01-02T11:00:00Z", "u2")
| ).toDF("timestamp", "userid").withColumn("curr_timestamp",unix_timestamp($"timestamp", "yyyy-MM-dd'T'HH:mm:ss'Z'").cast(TimestampType)).drop("timestamp")
clickstream: org.apache.spark.sql.DataFrame = [userid: string, curr_timestamp: timestamp]
scala>
scala> clickstream.show(false)
+------+-------------------+
|userid|curr_timestamp |
+------+-------------------+
|u1 |2018-01-01 11:00:00|
|u1 |2018-01-01 12:10:00|
|u1 |2018-01-01 13:00:00|
|u1 |2018-01-01 13:50:00|
|u1 |2018-01-01 14:40:00|
|u1 |2018-01-01 15:30:00|
|u1 |2018-01-01 16:20:00|
|u1 |2018-01-01 16:50:00|
|u2 |2018-01-01 11:00:00|
|u2 |2018-01-02 11:00:00|
+------+-------------------+
scala> //Generate column SEF with values 0 or 1 depending on whether difference between current and previous activity time is greater than 1 hour=3600 sec
scala>
scala> //Window on Current Timestamp when last activity took place
scala> val windowOnTs = Window.partitionBy("userid").orderBy("curr_timestamp")
windowOnTs: org.apache.spark.sql.expressions.WindowSpec = org.apache.spark.sql.expressions.WindowSpec#41dabe47
scala> //Create Lag Expression to find previous timestamp for the User
scala> val lagOnTS = lag(col("curr_timestamp"), 1).over(windowOnTs)
lagOnTS: org.apache.spark.sql.Column = lag(curr_timestamp, 1, NULL) OVER (PARTITION BY userid ORDER BY curr_timestamp ASC NULLS FIRST unspecifiedframe$())
scala> //Compute Timestamp for previous activity and subtract the same from Timestamp for current activity to get difference between 2 activities
scala> val diff_secs_col = col("curr_timestamp").cast("long") - col("prev_timestamp").cast("long")
diff_secs_col: org.apache.spark.sql.Column = (CAST(curr_timestamp AS BIGINT) - CAST(prev_timestamp AS BIGINT))
scala> val UserActWindowed=clickstream.withColumn("prev_timestamp", lagOnTS).withColumn("last_session_activity_after", diff_secs_col ).na.fill(0, Array("last_session_activity_after"))
UserActWindowed: org.apache.spark.sql.DataFrame = [userid: string, curr_timestamp: timestamp ... 2 more fields]
scala> //Generate Flag Column SEF (Session Expiry Flag) to indicate Session Has Expired due to inactivity for more than 1 hour
scala> val UserSessionFlagWhenInactive=UserActWindowed.withColumn("SEF",when(col("last_session_activity_after")>3600, 1).otherwise(0)).withColumn("tempsessid",sum(col("SEF")) over windowOnTs)
UserSessionFlagWhenInactive: org.apache.spark.sql.DataFrame = [userid: string, curr_timestamp: timestamp ... 4 more fields]
scala> UserSessionFlagWhenInactive.show(false)
+------+-------------------+-------------------+---------------------------+---+----------+
|userid|curr_timestamp |prev_timestamp |last_session_activity_after|SEF|tempsessid|
+------+-------------------+-------------------+---------------------------+---+----------+
|u1 |2018-01-01 11:00:00|null |0 |0 |0 |
|u1 |2018-01-01 12:10:00|2018-01-01 11:00:00|4200 |1 |1 |
|u1 |2018-01-01 13:00:00|2018-01-01 12:10:00|3000 |0 |1 |
|u1 |2018-01-01 13:50:00|2018-01-01 13:00:00|3000 |0 |1 |
|u1 |2018-01-01 14:40:00|2018-01-01 13:50:00|3000 |0 |1 |
|u1 |2018-01-01 15:30:00|2018-01-01 14:40:00|3000 |0 |1 |
|u1 |2018-01-01 16:20:00|2018-01-01 15:30:00|3000 |0 |1 |
|u1 |2018-01-01 16:50:00|2018-01-01 16:20:00|1800 |0 |1 |
|u2 |2018-01-01 11:00:00|null |0 |0 |0 |
|u2 |2018-01-02 11:00:00|2018-01-01 11:00:00|86400 |1 |1 |
+------+-------------------+-------------------+---------------------------+---+----------+
scala> //Compute Total session duration using the UDAF TotalSessionDuration such that :
scala> //(i)counter will be rest to 0 if SEF is set to 1
scala> //(ii)or set it to current session duration if session exceeds 2 hours
scala> //(iii)If both of them are inapplicable accumulate the sum
scala> val UserSessionDur=UserSessionFlagWhenInactive.withColumn("sessionSum",sessionSum(col("last_session_activity_after"),col("SEF")) over windowOnTs)
UserSessionDur: org.apache.spark.sql.DataFrame = [userid: string, curr_timestamp: timestamp ... 5 more fields]
scala> //Generate Session Marker if SEF is 1 or sessionSum Exceeds 2 hours(7200) seconds
scala> val UserNewSessionMarker=UserSessionDur.withColumn("SessionFlagChangeIndicator",when(col("SEF")===1 || col("sessionSum")>7200, 1).otherwise(0) )
UserNewSessionMarker: org.apache.spark.sql.DataFrame = [userid: string, curr_timestamp: timestamp ... 6 more fields]
scala> //Create New Session ID based on the marker
scala> val computeSessionId=UserNewSessionMarker.drop("SEF","tempsessid","sessionSum").withColumn("sessid",concat(col("userid"),lit("-"),(sum(col("SessionFlagChangeIndicator")) over windowOnTs)+1.toLong))
computeSessionId: org.apache.spark.sql.DataFrame = [userid: string, curr_timestamp: timestamp ... 4 more fields]
scala> computeSessionId.show(false)
+------+-------------------+-------------------+---------------------------+--------------------------+------+
|userid|curr_timestamp |prev_timestamp |last_session_activity_after|SessionFlagChangeIndicator|sessid|
+------+-------------------+-------------------+---------------------------+--------------------------+------+
|u1 |2018-01-01 11:00:00|null |0 |0 |u1-1 |
|u1 |2018-01-01 12:10:00|2018-01-01 11:00:00|4200 |1 |u1-2 |
|u1 |2018-01-01 13:00:00|2018-01-01 12:10:00|3000 |0 |u1-2 |
|u1 |2018-01-01 13:50:00|2018-01-01 13:00:00|3000 |0 |u1-2 |
|u1 |2018-01-01 14:40:00|2018-01-01 13:50:00|3000 |1 |u1-3 |
|u1 |2018-01-01 15:30:00|2018-01-01 14:40:00|3000 |0 |u1-3 |
|u1 |2018-01-01 16:20:00|2018-01-01 15:30:00|3000 |0 |u1-3 |
|u1 |2018-01-01 16:50:00|2018-01-01 16:20:00|1800 |1 |u1-4 |
|u2 |2018-01-01 11:00:00|null |0 |0 |u2-1 |
|u2 |2018-01-02 11:00:00|2018-01-01 11:00:00|86400 |1 |u2-2 |
+------+-------------------+-------------------+---------------------------+--------------------------+------+

Simple approach :
import spark.implicits._
val userActivity = Seq(
("2018-01-01 11:00:00", "u1"),
("2018-01-01 12:10:00", "u1"),
("2018-01-01 13:00:00", "u1"),
("2018-01-01 13:50:00", "u1"),
("2018-01-01 14:40:00", "u1"),
("2018-01-01 15:30:00", "u1"),
("2018-01-01 16:20:00", "u1"),
("2018-01-01 16:50:00", "u1"),
("2018-01-01 11:00:00", "u2"),
("2018-01-02 11:00:00", "u2")
).toDF("click_time", "user_id")
var df1 = userActivity.orderBy(asc("click_time")).
groupBy("user_id").agg(collect_list("click_time").alias("ts_diff"))
df1=df1.withColumn("test",convert_Case()($"ts_diff")).drop("ts_diff")
df1=df1.withColumn("test",explode(col("test"))).
withColumn("click_time",$"test._1").withColumn("session_generated",
concat($"user_id",lit("_"),$"test._2")).drop("test")
df1.show(truncate = false)
def convert_Case()=
udf {(timeList:Array[String])=>
import java.text.SimpleDateFormat
val dateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss")
val new_list = ListBuffer[(String, Int)]()
var session = 1
new_list.append((timeList(0), session))
var x = timeList(0)
var time = 0L
for(i<-1 until(timeList.length)){
val d1 = dateFormat.parse(x)
val d2 = dateFormat.parse(timeList(i))
val diff = ((d1.getTime - d2.getTime)/1000)*(-1)
val test = diff + time
if (diff >= 3600.0 || test >= 7200.0) {
session = session + 1
new_list.append((timeList(i), session))
time = 0L
}
else
{new_list.append((timeList(i), session))
time = test}
x = timeList(i)}
new_list
}

Complete solution
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.expressions.Window
import org.apache.spark.sql.functions._
import scala.collection.mutable.ListBuffer
import scala.util.control._
import spark.sqlContext.implicits._
import java.sql.Timestamp
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
val interimSessionThreshold=60
val totalSessionTimeThreshold=120
val sparkSession = SparkSession.builder.master("local").appName("Window Function").getOrCreate()
val clickDF = sparkSession.createDataFrame(Seq(
("2018-01-01T11:00:00Z","u1"),
("2018-01-01T12:10:00Z","u1"),
("2018-01-01T13:00:00Z","u1"),
("2018-01-01T13:50:00Z","u1"),
("2018-01-01T14:40:00Z","u1"),
("2018-01-01T15:30:00Z","u1"),
("2018-01-01T16:20:00Z","u1"),
("2018-01-01T16:50:00Z","u1"),
("2018-01-01T11:00:00Z","u2"),
("2018-01-02T11:00:00Z","u2")
)).toDF("clickTime","user")
val newDF=clickDF.withColumn("clickTimestamp",unix_timestamp($"clickTime", "yyyy-MM-dd'T'HH:mm:ss'Z'").cast(TimestampType).as("timestamp")).drop($"clickTime")
val partitionWindow = Window.partitionBy($"user").orderBy($"clickTimestamp".asc)
val lagTest = lag($"clickTimestamp", 1, "0000-00-00 00:00:00").over(partitionWindow)
val df_test=newDF.select($"*", ((unix_timestamp($"clickTimestamp")-unix_timestamp(lagTest))/60D cast "int") as "diff_val_with_previous")
val distinctUser=df_test.select($"user").distinct.as[String].collect.toList
val rankTest = rank().over(partitionWindow)
val ddf = df_test.select($"*", rankTest as "rank")
case class finalClick(User:String,clickTime:Timestamp,session:String)
val rowList: ListBuffer[finalClick] = new ListBuffer()
distinctUser.foreach{x =>{
val tempDf= ddf.filter($"user" === x)
var cumulDiff:Int=0
var session_index=1
var startBatch=true
var dp=0
val len = tempDf.count.toInt
for(i <- 1 until len+1){
val r = tempDf.filter($"rank" === i).head()
dp = r.getAs[Int]("diff_val_with_previous")
cumulDiff += dp
if(dp <= interimSessionThreshold && cumulDiff <= totalSessionTimeThreshold){
startBatch=false
rowList += finalClick(r.getAs[String]("user"),r.getAs[Timestamp]("clickTimestamp"),r.getAs[String]("user")+session_index)
}
else{
session_index+=1
cumulDiff = 0
startBatch=true
dp=0
rowList += finalClick(r.getAs[String]("user"),r.getAs[Timestamp]("clickTimestamp"),r.getAs[String]("user")+session_index)
}
}
}}
val dataFrame = sc.parallelize(rowList.toList).toDF("user","clickTimestamp","session")
dataFrame.show
+----+-------------------+-------+
|user| clickTimestamp|session|
+----+-------------------+-------+
| u1|2018-01-01 11:00:00| u11|
| u1|2018-01-01 12:10:00| u12|
| u1|2018-01-01 13:00:00| u12|
| u1|2018-01-01 13:50:00| u12|
| u1|2018-01-01 14:40:00| u13|
| u1|2018-01-01 15:30:00| u13|
| u1|2018-01-01 16:20:00| u13|
| u1|2018-01-01 16:50:00| u14|
| u2|2018-01-01 11:00:00| u21|
| u2|2018-01-02 11:00:00| u22|
+----+-------------------+-------+

-----Solution without using explode----.
`In my point of view explode is heavy process and inorder to apply you have taken groupby and collect_list.`
`
import pyspark.sql.functions as f
from pyspark.sql.window import Window
streaming_data=[("U1","2019-01-01T11:00:00Z") ,
("U1","2019-01-01T11:15:00Z") ,
("U1","2019-01-01T12:00:00Z") ,
("U1","2019-01-01T12:20:00Z") ,
("U1","2019-01-01T15:00:00Z") ,
("U2","2019-01-01T11:00:00Z") ,
("U2","2019-01-02T11:00:00Z") ,
("U2","2019-01-02T11:25:00Z") ,
("U2","2019-01-02T11:50:00Z") ,
("U2","2019-01-02T12:15:00Z") ,
("U2","2019-01-02T12:40:00Z") ,
("U2","2019-01-02T13:05:00Z") ,
("U2","2019-01-02T13:20:00Z") ]
schema=("UserId","Click_Time")
window_spec=Window.partitionBy("UserId").orderBy("Click_Time")
df_stream=spark.createDataFrame(streaming_data,schema)
df_stream=df_stream.withColumn("Click_Time",df_stream["Click_Time"].cast("timestamp"))
df_stream=df_stream\
.withColumn("time_diff",
(f.unix_timestamp("Click_Time")-f.unix_timestamp(f.lag(f.col("Click_Time"),1).over(window_spec)))/(60*60)).na.fill(0)
df_stream=df_stream\
.withColumn("cond_",f.when(f.col("time_diff")>1,1).otherwise(0))
df_stream=df_stream.withColumn("temp_session",f.sum(f.col("cond_")).over(window_spec))
new_window=Window.partitionBy("UserId","temp_session").orderBy("Click_Time")
new_spec=new_window.rowsBetween(Window.unboundedPreceding,Window.currentRow)
cond_2hr=(f.unix_timestamp("Click_Time")-f.unix_timestamp(f.lag(f.col("Click_Time"),1).over(new_window)))
df_stream=df_stream.withColumn("temp_session_2hr",f.when(f.sum(f.col("2hr_time_diff")).over(new_spec)-(2*60*60)>0,1).otherwise(0))
new_window_2hr=Window.partitionBy(["UserId","temp_session","temp_session_2hr"]).orderBy("Click_Time")
hrs_cond_=(f.when(f.unix_timestamp(f.col("Click_Time"))-f.unix_timestamp(f.first(f.col("Click_Time")).over(new_window_2hr))-(2*60*60)>0,1).otherwise(0))
df_stream=df_stream\
.withColumn("final_session_groups",hrs_cond_)
df_stream=df_stream.withColumn("final_session",df_stream["temp_session_2hr"]+df_stream["temp_session"]+df_stream["final_session_groups"]+1)\
.drop("temp_session","final_session_groups","time_diff","temp_session_2hr","final_session_groups")
df_stream=df_stream.withColumn("session_id",f.concat(f.col("UserId"),f.lit(" session_val----->"),f.col("final_session")))
df_stream.show(20,0)
`
---Steps taken to solve ---
` 1.first find out those clickstream which are clicked less than one hour and find the
continuous groups.
2.then find out the click streams based on the 2hrs condition and make the continuous groups for this condition to be apply i have to create two continuos group based on logic as follows .
3 . One group will be based on the sum of time difference and make one group i.e temp_session_2hr and based on that find the next group final_session_groups .
4.Sum of these above continuous groups and add +1 to populate the final_session column at the end of algo and do concat as per your requirement to show the session_id.`
result will be looking like this
`+------+---------------------+-------------+---------------------+
|UserId|Click_Time |final_session|session_id |
+------+---------------------+-------------+---------------------+
|U2 |2019-01-01 11:00:00.0|1 |U2 session_val----->1|
|U2 |2019-01-02 11:00:00.0|2 |U2 session_val----->2|
|U2 |2019-01-02 11:25:00.0|2 |U2 session_val----->2|
|U2 |2019-01-02 11:50:00.0|2 |U2 session_val----->2|
|U2 |2019-01-02 12:15:00.0|2 |U2 session_val----->2|
|U2 |2019-01-02 12:40:00.0|2 |U2 session_val----->2|
|U2 |2019-01-02 13:05:00.0|3 |U2 session_val----->3|
|U2 |2019-01-02 13:20:00.0|3 |U2 session_val----->3|
|U1 |2019-01-01 11:00:00.0|1 |U1 session_val----->1|
|U1 |2019-01-01 11:15:00.0|1 |U1 session_val----->1|
|U1 |2019-01-01 12:00:00.0|2 |U1 session_val----->2|
|U1 |2019-01-01 12:20:00.0|2 |U1 session_val----->2|
|U1 |2019-01-01 15:00:00.0|3 |U1 session_val----->3|
+------+---------------------+-------------+---------------------+
`

How to extract values from key value map?

I have a column of type map, where the key and value changes. I am trying to extract the value and create a new column.
Input:
----------------+
|symbols |
+---------------+
|[3pea -> 3PEA] |
|[barello -> BA]|
|[] |
|[] |
+---------------+
Expected output:
--------+
|symbols|
+-------+
|3PEA |
|BA |
| |
| |
+-------+
Here is what I tried so far using a udf:
def map_value=udf((inputMap:Map[String,String])=> {inputMap.map(x=>x._2)
})
java.lang.UnsupportedOperationException: Schema for type scala.collection.immutable.Iterable[String] is not supported

Since Spark scala v2.3 api, sql v2.3 api, or pyspark v2.4 api you can use the spark sql function map_values
The following is in pyspark, scala would be very similar.
Setup (assuming working SparkSession as spark):
from pyspark.sql import functions as F
df = (
spark.read.json(sc.parallelize(["""[
{"key": ["3pea"], "value": ["3PEA"] },
{"key": ["barello"], "value": ["BA"] }
]"""]))
.select(F.map_from_arrays(F.col("key"), F.col("value")).alias("symbols") )
)
df.printSchema()
df.show()
root
|-- symbols: map (nullable = true)
| |-- key: string
| |-- value: string (valueContainsNull = true)
+---------------+
| symbols|
+---------------+
| [3pea -> 3PEA]|
|[barello -> BA]|
+---------------+
df.select((F.map_values(F.col("symbols"))[0]).alias("map_vals")).show()
+--------+
|map_vals|
+--------+
| 3PEA|
| BA|
+--------+

import org.apache.spark.sql.functions._
import spark.implicits._
val m = Seq(Array("A -> abc"), Array("B -> 0.11856755943424617"), Array("C -> kqcams"))
val df = m.toDF("map_data")
df.show
// Simulate your data I think.
val df2 = df.withColumn("xxx", split(concat_ws("",$"map_data"), "-> ")).select($"xxx".getItem(1).as("map_val")).drop("xxx")
df2.show(false)
results in:
+--------------------+
| map_data|
+--------------------+
| [A -> abc]|
|[B -> 0.118567559...|
| [C -> kqcams]|
+--------------------+
+-------------------+
|map_val |
+-------------------+
|abc |
|0.11856755943424617|
|kqcams |
+-------------------+