Jan Schutte01/30/2023, 12:59 PM
on a partition of about 200GB, but my jobs often run out of memory. I've allocated
dt.optimize().where("partition == 'asd'").executeCompaction()
which I feel should be enough? Any ideas on how to optimize this operation for memory efficiency? I am not in a position to allocate more memory. I'm on Delta OSS 2.0.0 and Spark 3.2.2.
JosephK (exDatabricks)01/30/2023, 1:37 PM