Ryan Aston09/02/2023, 7:30 PM
Z-Ordering is not idempotent. Everytime the Z-Ordering is executed it will try to create a new clustering of data in all files (new and existing files that were part of previous Z-Ordering) in a partition.And here is a line from the Databricks docs:
Z-ordering is not idempotent but aims to be an incremental operation. The time it takes for Z-ordering is not guaranteed to reduce over multiple runs. However, if no new data was added to a partition that was just Z-ordered, another Z-ordering of that partition will not have any effect.Can someone confirm the behavior is different between OSS Delta and Databricks? Is there any way to do incremental Z-Ordering with delta-rs? Currently the tables in question do not have any partitioning, so without incremental Z-ordering it would be considering the entirety of the data set which can vary wildly (MBs to TBs). I’ve been considering adding a date partition to the tables in order to add a predicate to the Z-order operation but would prefer to avoid that as it doesn’t really fit the query pattern. Also, typical bin-packing mentions the file balancing is based around file size on disk whereas Z-ordering is focused more on number of tuples per file. Does this mean Z-ordering disregards the target file size? If so, how does it determine the division of files?
Will Jones09/03/2023, 12:59 AM
Can someone confirm the behavior is different between OSS Delta and Databricks?Hope this isn't too confusing, but delta-rs project is separate from the Spark OSS Delta, so we don't necessarily support all the same things they do. Right now z-order in delta-rs isn't an incremental operation and there isn't really a way to run it incrementally right now.
Does this mean Z-ordering disregards the target file size? If so, how does it determine the division of files?No we use target_file_size to determine where to split files, not row size.
Ryan Aston09/03/2023, 1:14 AM
Will Jones09/03/2023, 4:44 PM
Is standard bin-packing incremental in delta-rs?Yes. It's pretty easy to check if files are large enough that you can skip them.