We are facing a dilemma around how to improve the layout on disk of data which is going to be queried by using lte/gte filters on a decimal value. This sounds like a great use case for Z-Ordering optimizations in Delta Lake, but we are also concerned that this might result in an extraordinary amount of IO operations or shuffling, and we considered to create our own “bin” column and partition by bin. We haven’t found much on Delta Lake documentation that compares Z-Orders with explicit binning, does anyone have some useful insight on the topic?

can you explain more about your concerns with IO and shuffling? and how would users use the bin column? would they need to explicitly provide a predicate against the bin column?

That’s right, they would provide a predicate against the bin column

when it comes to z-ordering, those predicates and that type should be supported. maybe trial both binning and z-ordering on a subset of the data and see what happens?