Pyspark - Same filter as the predicate applied after scan even after the predicate is getting pushed down - sql

Question

Question: When joining two datasets, Why is the filter isnotnull applied twice on the joining key column? In the physical plan, it is once applied as a PushedFilter and then explicitly applied right after it. Why is that so?
code:
import os
import pandas as pd, numpy as np
import pyspark
spark=pyspark.sql.SparkSession.builder.getOrCreate()
save_loc = "gs://monsoon-credittech.appspot.com/spark_datasets/random_tests/"
df1 = spark.createDataFrame(pd.DataFrame({'a':np.random.choice([1,2,None],size = 1000, p = [0.47,0.48,0.05]),
'b': np.random.random(1000)}))
df2 = spark.createDataFrame(pd.DataFrame({'a':np.random.choice([1,2,None],size = 1000, p = [0.47,0.48,0.05]),
'b': np.random.random(1000)}))
df1.write.parquet(os.path.join(save_loc,"dfl_key_int"))
df2.write.parquet(os.path.join(save_loc,"dfr_key_int"))
dfl_int = spark.read.parquet(os.path.join(save_loc,"dfl_key_int"))
dfr_int = spark.read.parquet(os.path.join(save_loc,"dfr_key_int"))
dfl_int.join(dfr_int,on='a',how='inner').explain()
output:
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- Project [a#23L, b#24, b#28]
+- BroadcastHashJoin [a#23L], [a#27L], Inner, BuildRight, false
:- Filter isnotnull(a#23L)
: +- FileScan parquet [a#23L,b#24] Batched: true, DataFilters: [isnotnull(a#23L)], Format: Parquet, Location: InMemoryFileIndex[gs://monsoon-credittech.appspot.com/spark_datasets/random_tests/dfl_key_int], PartitionFilters: [], PushedFilters: [IsNotNull(a)], ReadSchema: struct<a:bigint,b:double>
+- BroadcastExchange HashedRelationBroadcastMode(List(input[0, bigint, false]),false), [id=#75]
+- Filter isnotnull(a#27L)
+- FileScan parquet [a#27L,b#28] Batched: true, DataFilters: [isnotnull(a#27L)], Format: Parquet, Location: InMemoryFileIndex[gs://monsoon-credittech.appspot.com/spark_datasets/random_tests/dfr_key_int], PartitionFilters: [], PushedFilters: [IsNotNull(a)], ReadSchema: struct<a:bigint,b:double>

Answer 1

The reason is that a PushedFilter does not guarantee you that all the data is filtered as you want before it has been read into memory by Spark. For more context on what a PushedFilter is, check out this SO answer.
Parquet files
Let's have a look at Parquet files like in your example. Parquet files are stored in a columnar format, and they are also organized in Row Groups (or chunks). The following picture comes from the Apache Parquet docs:
You see that the data is stored in a columnar fashion, and they are chopped up into chunks (row groups). Now, for each column/row chunk combination, Parquet stores some metadata. In that picture, you see that it contains a bunch of metadata and then also extra key/value pairs. These also contain statistics about your data (depending on what type your column is).
Some examples of these statistics are:
what the min/max value is of the chunk (in case it makes sense for the data type of the column)
whether the chunk has non-null values
...
Back to your example
You are joining on the a column. To be able to do that we need to be sure that a has no null values. Let's imagine that your a column (disregarding the other columns) is stored like this:
a column:
chunk 1: 0, 1, None, 1, 1, None
chunk 2: 0, 0, 0, 0, 0, 0
chunk 3: None, None, None, None, None, None
Now, using the PushedFilter we can immediately (just by looking at the metadata of the chunks) disregard chunk 3, we don't even have to read it in!
But as you see, chunk 1 still contains null values. This is something we can't filter out by only looking at the chunk's metadata. So we'll have to read in that whole chunk and then filter those other null values afterwards within Spark using that second Filter node in your Physical Plan.