Spark SQL Correlated scalar sub-queries can only be used in a Filter/Aggregate/Project - apache-spark-sql

Question

I am trying to join tow streaming data in Spark structured streaming. Data structures are as follows:
Table: CardHolder
CardNo
AssignTime
Assignee
1
01/01/2023 01:00:00
user 1
1
01/01/2023 01:00:05
user 2
1
01/01/2023 01:00:10
user 3
Table: CardTransaction
CardNo
TransactionTime
Transaction
1
01/01/2023 01:00:00
transaction 1
1
01/01/2023 01:00:10
transaction 2
1
01/01/2023 01:00:03
transaction 3
1
01/01/2023 01:00:07
transaction 4
1
01/01/2023 01:00:12
transaction 5
Those two data should be joined into this data:
Table: CardTransactionFlow
CardNo
TransactionTime
Transaction
Assignee
1
01/01/2023 01:00:00
transaction 1
user 1
1
01/01/2023 01:00:10
transaction 2
user 3
1
01/01/2023 01:00:03
transaction 3
user 1
1
01/01/2023 01:00:07
transaction 4
user 2
1
01/01/2023 01:00:12
transaction 5
user 3
The issue is that TransactionTime in CardTransaction table should match with the latest AssignTime in CardHolder table before transactionTime for earch transaction record. To archive this i tried to use max(assignTime) function but i can not get it to work. Spark throws org.apache.spark.sql.AnalysisException: Correlated scalar sub-queries can only be used in a Filter/Aggregate/Project and a few commands: Join LeftOuter. Watermark ise set on date columns.
select
holder.cardNo,
trx.transactionTime,
trx.transaction,
holder.assignee,
from transaction trx LEFT OUTER JOIN holder holder ON
trx.cardNo=holder.cardNo AND
holder.assignTime = (select max(assignTime) from holder
where trx.cardNo=cardNo AND assignTime <= trx.transactionTime) + interval 1 minute
Any help would be appreciated.

Answer 1

Use the lead functnion for the df1 to get the time range and join with df2.
data1 = [
[1, '01/01/2023 01:00:00', 'user 1'],
[1, '01/01/2023 01:00:05', 'user 2'],
[1, '01/01/2023 01:00:10', 'user 3']
]
data2 = [
[1, '01/01/2023 01:00:00', 'transaction 1'],
[1, '01/01/2023 01:00:10', 'transaction 2'],
[1, '01/01/2023 01:00:03', 'transaction 3'],
[1, '01/01/2023 01:00:07', 'transaction 4'],
[1, '01/01/2023 01:00:12', 'transaction 5']
]
df1 = spark.createDataFrame(data1, ['CardNo', 'AssignTime', 'Assignee']).withColumn('AssignTime', f.to_timestamp('AssignTime', 'dd/MM/yyyy hh:mm:ss'))
df2 = spark.createDataFrame(data2, ['CardNo', 'TransactionTime', 'Transaction']).withColumn('TransactionTime', f.to_timestamp('TransactionTime', 'dd/MM/yyyy hh:mm:ss'))
df1.printSchema()
w = Window.partitionBy('CardNo').orderBy('AssignTime')
df3 = df1.withColumn('AssignTimeAfter', f.lead('AssignTime', 1, '2999-12-31 23:59:59').over(w))
df3.show()
df2.alias('a') \
.join(df3.alias('b'), (f.col('a.CardNo') == f.col('b.CardNo')) & (f.col('a.TransactionTime') >= f.col('b.AssignTime')) & (f.col('a.TransactionTime') < f.col('b.AssignTimeAfter')), 'left') \
.select('a.*', 'b.Assignee') \
.show()
root
|-- CardNo: long (nullable = true)
|-- AssignTime: timestamp (nullable = true)
|-- Assignee: string (nullable = true)
+------+-------------------+--------+-------------------+
|CardNo| AssignTime|Assignee| AssignTimeAfter|
+------+-------------------+--------+-------------------+
| 1|2023-01-01 01:00:00| user 1|2023-01-01 01:00:05|
| 1|2023-01-01 01:00:05| user 2|2023-01-01 01:00:10|
| 1|2023-01-01 01:00:10| user 3|2999-12-31 23:59:59|
+------+-------------------+--------+-------------------+
+------+-------------------+-------------+--------+
|CardNo| TransactionTime| Transaction|Assignee|
+------+-------------------+-------------+--------+
| 1|2023-01-01 01:00:00|transaction 1| user 1|
| 1|2023-01-01 01:00:10|transaction 2| user 3|
| 1|2023-01-01 01:00:03|transaction 3| user 1|
| 1|2023-01-01 01:00:07|transaction 4| user 2|
| 1|2023-01-01 01:00:12|transaction 5| user 3|
+------+-------------------+-------------+--------+