Is there a way to force

deltaTable.optimize().executeZOrderBy

to sort records within each of the Parquet files according to the sort column? At the moment it seems like the data within each file is not sorted by the sort column and this effects our file sizes as similar records are not located next to each other

What do you mean by sort column?

Prior using z-ordering we were having our own implemented compaction which ordered the data within each Parquet file (using sort prior write). This made similar records get closer in the Parquet file and helped compaction, reduced parquet file size . Now, switching to delta 2, we started to use zordering instead of our own compaction but we seemed to have lost the ordering within the files themselves, resulting in larger Parquet files and an overall larger table size

I tried looking around and I found a . There is a sortWithinPartitions config that you can set to true, but I’m not certain that it’s used for delta and I’m not certain it’s used for delta zordering

Thanks, Looking at the zordering optimizer code, it seems to be doing

repartitionByRange

so no sorting option if I understand correctly. https://github.com/delta-io/delta/blob/master/core/src/main/scala/org/apache/spark/sql/delta/skipping/MultiDimClustering.scala#L50