Hey team, I am executing

dt.optimize().where("partition == 'asd'").executeCompaction()

on a partition of about 200GB, but my jobs often run out of memory. I've allocated

"spark.executor.memory": "45G"

and

"spark.executor.memoryOverhead": "5G"

which I feel should be enough? Any ideas on how to optimize this operation for memory efficiency? I am not in a position to allocate more memory. I'm on Delta OSS 2.0.0 and Spark 3.2.2.

Tuning.jpg