how do i transform data in sql or in pyspark - sql

Question

input dataset
Act status from to
123 1 2011-03-29 00:00:00 2011-03-29 23:59:59
123 1 2011-03-30 00:00:00 2011-03-30 23:59:59
123 1 2011-03-31 00:00:00 2011-03-31 23:59:59
123 0 2011-04-01 00:00:00 2011-04-03 23:59:59
123 0 2011-04-04 00:00:00 2011-04-04 23:59:59
123 0 2011-04-05 00:00:00 2011-04-05 23:59:59
123 1 2011-04-06 00:00:00 2011-04-06 23:59:59
123 1 2011-04-07 00:00:00 2011-04-07 23:59:59
123 1 2011-04-08 00:00:00 2011-04-10 23:59:59
I want output to be
act status from to
123 1 2011-03-29 00:00:00 2011-03-31 23:59:59
123 0 2011-04-01 00:00:00 2011-04-05 23:59:59
123 1 2011-04-06 00:00:00 2011-04-10 23:59:59

Answer 1

You can use the lag function to track the status change. After applying the lag function, you would use the results to build your rankings and use the rankings as your groupBy parameter. For example:
status
lag
changed
rankings
1
null
1
1
1
1
0
1
0
1
1
2
1
0
1
3
0
1
1
4
1
0
1
5
1
1
0
5
where:
status : current status
lag : status from previous row
changed : 0 if status == lag, otherwise 0
rankings : cumulative sum of changed
Anyway, here's the answer to your question in SQL.
spark.sql('''
SELECT Act, status, from, to
FROM (
SELECT Act, status, MIN(from) AS from, MAX(to) AS to, rankings
FROM (
SELECT *, SUM(changed) OVER(ORDER BY from) AS rankings
FROM (
SELECT *, IF(LAG(status) OVER(PARTITION BY Act ORDER BY from) = status, 0, 1) AS changed
FROM sample
)
)
GROUP BY 1,2,5
)
''').show()
output:
+---+------+-------------------+-------------------+
|Act|status| from| to|
+---+------+-------------------+-------------------+
|123| 1|2011-03-29 00:00:00|2011-03-31 23:59:59|
|123| 0|2011-04-01 00:00:00|2011-04-05 23:59:59|
|123| 1|2011-04-06 00:00:00|2011-04-10 23:59:59|
+---+------+-------------------+-------------------+
And, here's the pyspark version:
from pyspark.sql.functions import lag, sum, max, min, when
from pyspark.sql.window import Window
df2 = df.withColumn("lag_tracker",lag("status",1).over(Window.partitionBy("Act").orderBy("from")))
df2 = df2.withColumn("changed", when(df2.lag_tracker == df2.status, 0).otherwise(1))
df2 = df2.withColumn("rankings", sum("changed").over(Window.orderBy("from")))
df2 = df2.groupBy("Act", "status", "rankings").agg(min("from").alias("from"), max("to").alias("to"))
df2.select("Act", "status", "from", "to").show()
output:
+---+------+-------------------+-------------------+
|Act|status| from| to|
+---+------+-------------------+-------------------+
|123| 1|2011-03-29 00:00:00|2011-03-31 23:59:59|
|123| 0|2011-04-01 00:00:00|2011-04-05 23:59:59|
|123| 1|2011-04-06 00:00:00|2011-04-10 23:59:59|
+---+------+-------------------+-------------------+

Answer 2

If you have no gaps in the dates, I would suggest uses the difference of row numbers:
select act, status, min(from), max(to)
from (select t.*,
row_number() over (partition by act order by from) as seqnum,
row_number() over (partition by act, status order by from) as seqnum_2
from t
) t
group by act, status, (seqnum - seqnum_2);
Why this works is a little tricky to explain. However, if you look at the results from the subquery, you will see that the difference between seqnum and seqnum_2 is constant on adjacent rows with the same status.
Note: I would advise you to fix your data model so you don't miss the last second on each day. The to datetime of one row should be the same as the from datetime of the next row. When querying, you can use >= and < to get the values that the row matches.