Hey everyone I have some questions about Z Ordering in delta

Question

Hey everyone I have some questions about Z Ordering in delta rs For context is a line from the OSS Delta docs gt Z Ordering is not idempotent Everytime the Z Ordering is executed it will try to create a new clustering of data in all files new and existing files that were part of previous Z Ordering in a partition And is a line from the Databricks docs gt Z ordering is not idempotent but aims to be an incremental operation The time it takes for Z ordering is not guaranteed to reduce over multiple runs However if no new data was added to a partition that was just Z ordered another Z ordering of that partition will not have any effect Can someone confirm the behavior is different between OSS Delta and Databricks Is there any way to do incremental Z Ordering with delta rs Currently the tables in question do not have any partitioning so without incremental Z ordering it would be considering the entirety of the data set which can vary wildly MBs to TBs I ve been considering adding a date partition to the tables in order to add a predicate to the Z order operation but would prefer to avoid that as it doesn t really fit the query pattern Also typical bin packing mentions the file balancing is based around file size on disk whereas Z ordering is focused more on number of tuples per file Does this mean Z ordering disregards the target file size If so how does it determine the division of files

Will Jones · Answer

gt Can someone confirm the behavior is different between OSS Delta and Databricks Hope this isn t too confusing but delta rs project is separate from the Spark OSS Delta so we don t necessarily support all the same things they do Right now z order in delta rs isn t an incremental operation and there isn t really a way to run it incrementally right now

Will Jones · Answer

gt Does this mean Z ordering disregards the target file size If so how does it determine the division of files No we use target file size to determine where to split files not row size

Ryan Aston · Answer

Hi Will Understood delta rs doesn t have full feature parity with the Spark implementation and thank you for clarifying on the lack of incremental Z order support Is standard bin packing incremental in delta rs With regards to target file size I was looking at the source code and noticed the target file size argument is not passed to the merge plan for Z ordering only for bin packing Where is the target file size determined set for Z order <https github com delta io delta rs blob 787c13a63efa9ada96d303c10c093424215aaa80 rust src operations optimize rs L740|https github com delta io delta rs blob 787c13a63efa9ada96d303c10c093424215aaa80 rust src operations optimize rs L740>

Ryan Aston · Answer

Ah never mind on the last question Makes sense the target file size wouldn t be relevant for the Z order merge plan since it isn t incremental I do see in the source the compaction merge plan is skipping files already at the target file size and the target size is then added to the input params

Will Jones · Answer

Yeah it s not relevant to planning z order just writing out z order

Will Jones · Answer

The `target file size` is passed to the writer config here <https github com delta io delta rs blob 787c13a63efa9ada96d303c10c093424215aaa80 rust src operations optimize rs L421>

Will Jones · Answer

gt Is standard bin packing incremental in delta rs Yes It s pretty easy to check if files are large enough that you can skip them