More efficiently writing partitioned parquet when partitioning column is skewed - sql

Question

I'm working on writing a large table (approximately 1.2b rows) in partitioned parquet, I'm using state (like US state) as the partitioning key. The issue is that there is a large number of null state values. This table is often queried by state, so having a large partition with the null states is not an issue, but I'm having trouble more efficiently generating the table.
I've tried creating the table with the non-null states, then inserting the null, but from what I can tell all the null values still just get put in one big partition and therefore sent to one worker.
It would be great if there was a way to insert into a specific partition. Like for my example, write the non-null states, then insert remaining records into the state=null or hive_default_partition in a way that would still parallelize across the cluster.

Answer 1

Try writing the non-null data using automatic partitioning, then repartition the null data and write it separately, e.g.:
df.where($”state”.isNotNull).write.partitionBy($”state”).parquet(“my_output_dir”)
df.where($”state”.isNull).repartition(100).write.parquet(“my_output_dir/state=__HIVE_DEFAULT_PARTITION__”)
Using the SQL API, you can use a repartitioning hint (introduced in Spark 2.4) to accomplish the same:
spark-sql> describe skew_test;
id bigint NULL
dt date NULL
state string NULL
# Partition Information
# col_name data_type comment
state string NULL
Time taken: 0.035 seconds, Fetched 6 row(s)
spark-sql> CREATE TABLE `skew_test2` (`id` BIGINT, `dt` DATE, `state` STRING)
> USING parquet
> OPTIONS (
> `serialization.format` '1'
> )
> PARTITIONED BY (state);
Time taken: 0.06 seconds
spark-sql> insert into table skew_test2 select * from skew_test where state is not null;
Time taken: 1.208 seconds
spark-sql> insert into table skew_test2 select /*+ REPARTITION(100) */ * from skew_test where state is null;
Time taken: 1.39 seconds
You should see 100 tasks created by Spark for the final statement, and your state=__HIVE_DEFAULT_PARTITION__ directory should contain 100 parquet files. For more information on Spark-SQL hints, check out https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-hint-framework.html#specifying-query-hints