Best Practices for Spark SQL Temp View - hive

Question

I am new to Spark. We use Spark-SQL to query Hive tables on AWS EMR.
I am running a complex query by building several temporary views in steps.
For e.g. the first temp view is created by joining couple of tables in Step 1, then using this temp view
as a source in the next Step, and so on, until the final Step, where the resultant of the final Step is persisted to disk. An example is given below:
create temporary view test1 as
select a.cust_id, b.prod_nm
from a
inner join b
on a.id = b.id;
create temporary view test2 as
select t1.*, t2.*
from test1 t1
inner join c t2
on t1.cust_id = t2.cust_id;
Please note that the resultant view (test1) from the first step is used in the second step as a source in the Join with another table C.
Now, due to the lazy evaluation of Spark, even though the temp views get created at each step, the data is not pulled until the last step. As a result, we often run into performance issues in the queries that implement complex transformations (eg. join on several tables).
Basically I have 2 questions:
How to approximate the size of such a temp view (at any given step)
so that I can choose the right Join strategy in the next Step when
this view is Joined to another table/view in the next Step
What are the best practices to use for such a framework to
ameliorate performance.
Note: We use Spark 2.4. I do not have access to PySpark, but only access to SparkSQL (to query hive tables).
Any help is appreciated. Thanks.

Answer 1

You cannot determine the size of a temp view that you create.
When using a distributed framework like spark, your join strategy shouldn't be based on the size of data but how the data/join keys are distributed across the multiple partitions. If you are using the same temp view multiple times, it's better to cache it, so that the application doesn't read it from hdfs/s3 every time.
code to cache in SQL CACHE TABLE cache_view AS
SELECT * from table;