Spark Scala, write data with SaveMode.Append while overwriting some existing partition - dataframe

Question

I have the following command, in my spark program
df.write
.mode(SaveMode.Append)
.partitionBy("year","month","day")
.format(format)
.option("path",path)
.saveAsTable(table_name)
When I run it twice on the same date, I have duplicates in my data. So i want it to append the data but when some partitions already exist, it should overwrite them.

Answer 1

Here using Hive Integration but can be Spark only catalog, a complete example:
1) Need to set up table
spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")
import org.apache.spark.sql.types._
val df = spark.range(9).map(x => (x, (x + 100) % 3)).toDF("c1", "c2")
df.repartition($"c2")
.write
.partitionBy("c2")
.mode("overwrite").saveAsTable("tabX")
2) Update a partition - contrived, in this way after setup
val df2 = spark.range(1).map(x => (x, (x + 100) % 3)).toDF("c1", "c2")
df2.repartition($"c2")
.write
.mode("overwrite").insertInto("tabX")
3) Look at effects:
// from 9 -> 7 entries, pls run
val df3 = spark.table("tabX")
df3.show(false)
returns:
+---+---+
|c1 |c2 |
+---+---+
|2 |0 |
|5 |0 |
|8 |0 |
|1 |2 |
|4 |2 |
|7 |2 |
|0 |1 |
+---+---+
Here is the evidence of partition overwrite, then.

Answer 2

This can be achieved in 2 steps:
add the following spark conf,
sparkSession.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")
I used the following function to deal with the cases where I should overwrite or just append.
first import org.apache.hadoop.fs.{FileSystem, Path}
def writePartitionOverriding(df: DataFrame, tableName: String, path: String, spark: SparkSession): Unit = {
val fullPath = path + partition
if (FileSystem.get(spark.sparkContext.hadoopConfiguration).exists(new Path(fullPath)))
dfToWrite.write
.mode(SaveMode.Overwrite)
.format("orc")
.partitionBy("year","month","day")
.save(path)
else
writePartitionBySaveAsTable(dfToWrite, tableName, path, partition)
}
def writePartitionBySaveAsTable(df: DataFrame, tableName: String, path: String): Unit=
df.write
.mode(SaveMode.Append)
.format("orc")
.partitionBy("year","month","day")
.option("path", path)
.saveAsTable(tableName)