Py spark join on pipeline separated column - apache-spark-sql

Question

I have two data frames which i want to join. The catch is the one of the tables have pipeline separated string on which one of the value is what I want to join with. How do I it in Pyspark. Below is an example
TABLE A has
+-------+--------------------+
|id | name |
+-------+--------------------+
| 613760|123|test|test2 |
| 613740|456|ABC |
| 598946|OMG|567 |
TABLE B has
+-------+--------------------+
|join_id| prod_type|
+-------+--------------------+
| 123 |Direct De |
| 456 |Direct |
| 567 |In |
Expected Result - Join table A and Table B when there is a match with Table A's pipeline separated ID against Table B's value. For instance TableA.id - 613760 the name has 123|test and I want to join with table B's join ID 123 likewise 456 and 567.
Resultant Table
+--------------------+-------+
| name |join_Id|
+-------+------------+-------+
|123|test|test2 |123 |
|456|ABC |456 |
|OMG|567 |567 |
Can someone help me solve this. I am relatively new to pyspark and I am learning

Answer 1

To solve your problem you need to:
split those "pipeline separated strings"
then exploding those values into separated rows.
posexplode would do that for you http://spark.apache.org/docs/2.1.0/api/python/pyspark.sql.html#pyspark.sql.functions.posexplode
from there an "inner join" and
finally a "select" would do the rest of the trick.
See the code below:
import pyspark.sql.functions as f
#First create the dataframes to test solution
table_A = spark.createDataFrame([(613760, '123|test|test2' ), (613740, '456|ABC'), (598946, 'OMG|567' )], ["id", "name"])
# +-------+--------------------+
# |id | name |
# +-------+--------------------+
# | 613760|123|test|test2 |
# | 613740|456|ABC |
# | 598946|OMG|567 |
table_B = spark.createDataFrame([('123', 'Direct De' ), ('456', 'Direct'), ('567', 'In' )], ["join_id", "prod_type"])
# +-------+--------------------+
# |join_id| prod_type|
# +-------+--------------------+
# | 123 |Direct De |
# | 456 |Direct |
# | 567 |In |
result = table_A \
.select(
'name',
f.posexplode(f.split(f.col('name'),'\|')).alias('pos', 'join_id')) \
.join(table_B, on='join_id', how='inner') \
.select('name', 'join_id')
result.show(10, False)
# +--------------+-------+
# |name |join_id|
# +--------------+-------+
# |123|test|test2|123 |
# |456|ABC |456 |
# |OMG|567 |567 |
# +--------------+-------+
Hope that works. As you continue getting better in Pyspark. I would recommend you to go through the functions in pyspark.sql.functions and that would take your skills to the next level.