Translate SQL UPDATE with multiple INNER JOIN and WHERE into PySpark - sql

Question

I am trying to execute the below code using PySpark:
join_on = (df_1.C1_PROFIT == df_2.C2_PROFIT) & \ # JOIN CONDITION
(df_1.C1_REVENUE == df_3.C3_REVENUE_BREAK) & \ # JOIN CONDITION
(df_1.C1_LOSS == df_4.C4_TOTAL_LOSS) & \ # JOIN CONDITION
((df_4.TOTAL_YEAR_PROFIT) > (df_3.TOTAL_GROWTH)) # WHERE CONDITION
df = (df_1.alias('a')
.join(df_2.alias('b'), join_on, 'left')
.join(df_3.alias('c'), join_on, 'left')
.join(df_4.alias('d'), join_on, 'left')
.select(
*[c for c in df_2.columns if c != 'C2_TARGET'],
F.expr("nvl2(b.C2_PROFIT, '500', a.C2_TARGET) C2_TARGET")
)
)
Error after running the query:
'TOTAL_YEAR_PROFIT','TOTAL_GROWTH', 'TOTAL_LOSS', 'REVENUE_BREAK'
does not exist in df_1 columns:
The original SQL query:
UPDATE (( companyc1
INNER JOIN companyc2
ON company1.c1_profit = company2.c2_profit)
INNER JOIN companyc3
ON company1.c1_revenue = company3.revenue_break)
INNER JOIN companyc4
ON company1.c1_loss = company4.c4_total_loss
SET companyc1.sales = "500"
WHERE (( ( company4.total_year_profit ) > [company3].[total_growth] ))
Can anyone help me find where I am doing the mistake?

Answer 1

Your join_on conditions will have to be split for each of the join operation, like so:
df = (df_1.alias('a')
.join(df_2.alias('b'), df_1.C1_PROFIT == df_2.C2_PROFIT, 'left')
.join(df_3.alias('c'), df_1.C1_REVENUE == df_3.C3_REVENUE_BREAK, 'left')
.join(df_4.alias('d'), df_1.C1_LOSS == df_4.C4_TOTAL_LOSS. 'left')
.select(
*[c for c in df_2.columns if c != 'C2_TARGET'],
F.expr("nvl2(b.C2_PROFIT, '500', a.C2_TARGET) C2_TARGET")
).where("d.TOTAL_YEAR_PROFIT > c.TOTAL_GROWTH")
)

Answer 2

When translating SQL UPDATE containing multiple joins, it seems to me, that the universally safe approach could involve groupBy, agg and monotonically_increasing_id (to make sure that the row number of the original df will not shrink after the aggregation).
I've made the following tables in MS Access, to make sure that the approach I suggest, would work the same way in Spark.
Inputs:
Result after the update:
Spark
It seems, MS Access aggregates column values, so the following will do the same.
Inputs:
from pyspark.sql import functions as F
df_1 = spark.createDataFrame(
[(2, 10, 5, 'replace'),
(2, 10, 5, 'replace'),
(1, 10, None, 'keep'),
(2, None, 5, 'keep')],
['C1_PROFIT', 'C1_REVENUE', 'C1_LOSS', 'SALES']
)
df_2 = spark.createDataFrame([(1,), (2,), (8,)], ['C2_PROFIT'])
df_3 = spark.createDataFrame([(10, 51), (10, 50)], ['REVENUE_BREAK', 'TOTAL_GROWTH'])
df_4 = spark.createDataFrame([(5, 50), (5, 51),], ['C4_TOTAL_LOSS', 'TOTAL_YEAR_PROFIT'])
Script:
df_1 = df_1.withColumn('_id', F.monotonically_increasing_id())
df = (df_1.alias('a')
.join(df_2.alias('b'), df_1.C1_PROFIT == df_2.C2_PROFIT, 'left')
.join(df_3.alias('c'), df_1.C1_REVENUE == df_3.REVENUE_BREAK, 'left')
.join(df_4.alias('d'), df_1.C1_LOSS == df_4.C4_TOTAL_LOSS, 'left')
.groupBy(*[c for c in df_1.columns if c != 'SALES'])
.agg(F.when(F.max('d.total_year_profit') > F.min('c.total_growth'), '500')
.otherwise(F.first('a.SALES')).alias('SALES')
).drop('_id')
)
df.show()
# +---------+----------+-------+-----+
# |C1_PROFIT|C1_REVENUE|C1_LOSS|SALES|
# +---------+----------+-------+-----+
# | 1| 10| null| keep|
# | 2| null| 5| keep|
# | 2| 10| 5| 500|
# | 2| 10| 5| 500|
# +---------+----------+-------+-----+