Kevin Lim

09/18/2023, 7:12 AM
Anyone know if liquid clustering will be supported (or is supported) in delta-rs?

Eero Lihavainen

09/18/2023, 7:18 AM
Looks like support for reading databricks liquid clustered tables is upcoming: But as for the clustering itself it's not part of the delta spec yet

Will Jones

09/18/2023, 4:54 PM
I think we can't start on liquid clustering until a design doc is released with details. I don't think OSS Spark delta lake has this yet (but correct me if I'm wrong)

Kevin Lim

09/18/2023, 7:38 PM
Ah, I see, I wasn't aware that it wasn't open yet.


09/18/2023, 9:50 PM
Yeah, it's available in DBR 13.x from Databricks but not yet part of the upcoming Delta 3.0.0 release as far as I understand it. I am still quite curious as to the implementation since there's some coordination between the read and write side of the equation to make it work 🤔