Joining multiple spark dataframes on non-primary key - dataframe

Question

I am trying to join 1 parent Dataframe with 2 child Dataframes.
Here is how my parent DF look like
PersonId
FirstName
LastName
1
ABC
XYZ
Child DF 1
FirstName
FirstNameMatchedPersonIds
ABC
[1, 10, 20]
Child DF2
LastName
LastNameMatchedPersonIds
XYZ
[1, 40, 70]
I want to join parent DF and child DF 1 by FirstName column and the result should be joined with child DF 2 by LastName column. Here is how my expected join result should look like
PersonId
FirstName
LastName
FirstNamePersonIds
LastNamePersonIds
1
ABC
XYZ
[1,10,20]
[1,40,70]
What will be a good join strategy to achieve this result? Simple inner join is resulting in lot of shuffle writes and often job fails after running for a long time.
Some Estimations:
Number of records in Parent DF : 35 million
Number of records in Child DF1 : 1.5 million
Number of records in Child DF2 : 1.5 million
I would have more such child DF's (at least 15) which I would join with different columns in the parent DF
Additional info: Source for my parent DF is Hbase table which I scan and create RDD and write to sequence file. Further more I read the sequence files and create the Dataframes (so that Hbase scan is done only once).

Answer 1

For me those two smaller datasets are good candidates for broadcast join, you may try to use it and check the results. Broadcast join
Please remember that broadcast should be used carefully and only on relatively small datasets. Your broadcasted dataset will be collected on driver (so it needs to fit in its memory) and then propagated to other executors so size is really important here
If broadcast is not an option here you may check if you have data skew and if you are using Spark 3.X you may turn AQE on and check if it helps

Answer 2

Many a times, we cannot tweak joins more than a certain amount. We have to join to achieve the desired results.
What we can rather try to do is, calibrate the spark.sql.shuffle.partitions config based on the size/partitions of the data and the total number of executor cores available for your Spark application. Default value for this is 200 which in most cases prove to be too less. Ideally, I would recommend you to set this in multiples of the executor cores. For e.g., if you have 100 executors and 4 cores each, start with 800 and calibrate this based on the findings from Spark UI.
Hope this helps!