Writing a huge dataframe iterator to S3 Any configuration to reclaim memory after write - apache-spark-sql

Question

I have this method that writes huge dataframe to S3 and this runs in a docker which has been allocated 30 GB memory but this is giving a OOM in the node and the task dies. Is there any spark configuration that can be set and is there any efficient way to do this. This cardsWithdfs can have 9 frames or frames up to 72. I want to know why the memory keeps going up even when i am processing only two dataframes and how can i do this clean up operation to claim the memory back. The dataframe can be 3GB in size
def writeParquetData(basePath: String, batchNumber: Int, resultCatalogCard: ResultsCatalogCard, cardsWithdfs: Iterable[(Int, DataFrame, String)], errorCategory: String, sortColumn: String, partitionColumns: Seq[String]): Iterable[(Int, String, Boolean, String)] = {
var outLevelIterator: mutable.MutableList[(Int, String, Boolean, String)] = mutable.MutableList.empty
cardsWithdfs.grouped(2).foreach( batch => {
val writeDataFrames = Future.traverse(batch) {
cardDataFrameCodeTriple =>
Future {
val (outputLevelCode, df, perspCode) = cardDataFrameCodeTriple
//.....
Try {
if (!df.isEmpty) {
saveResultsSortedDesc(df, partitionColumns, Seq(sortColumn), ulfPathLocation)
(outputLevelCode, pathWithouttenant, false, perspCode)
} else (outputLevelCode, pathWithouttenant, true, perspCode) // df is empty path does not matter
} match {
case Success(x) => {
val (outLevel, ulfPath, ignoreSchema, perspective) = x
logger.info(s"RUN: writeParquetData ulfPathLocation written for workflowId: ${workflowId}, analysisId: ${analysisId} outputLevel: ${outputLevelCode} lossPath: ${ulfPath} ignoreSchema ${ignoreSchema},perspective ${perspective}")
//Add to iterator
}
case Failure(ex) => this.synchronized {
resultsMetadataClient.update(resultCatalogCard.id.toInt, s"""{"$errorCategory": "FAILED", "$CONVERTED_TO_ULF_MIGRATION_ERROR": "FAILED - analysisId: ${analysisId} runningWorkflowId: $runningWorkflowId with ${ex.getMessage}"}""")
logger.warn(s"Exception Analysis readParquetData df for workflowId: ${workflowId}, analysisId: ${analysisId} outputLevel: ${outputLevelCode} batchNumber: ${batchNumber} failed with ${ex.getMessage} StackTrace: ${ex.printStackTrace()}", ex)
}
}
}
}
Await.result(writeDataFrames, Duration.Inf)
})
outLevelIterator
}