Just tested liquid clustering vs partitioning on a 1 PB tabl

Question

Just tested liquid clustering vs partitioning on a 1 PB table Results are Extraordinary to say the least right now This was just a test table and the final table will prob be about 200PB but will feed back to the product group my findings Simply amazed so far

rtyler · Answer

dannnng that s a big table

Dominique Brezinski · Answer

Good to know I have a great candidate table for liquid that will be in that size range Encouraging

Jordan Fox · Answer

Results were extraordinary but what were the results

Felipe Pessoto · Answer

How did you test it Is it in 3 0 0 RC1

Dominique Brezinski · Answer

Would be in Databricks Runtime 13

Barny Self · Answer

DBR 13 2 Having some unexpected results so far so am going to talk to the team before sharing anything further because it is probably a me being an idiot situation rolling on the floor laughing

Venkat Hari · Answer

< Barny Self> any updates on this My team and I are seeing read operation is performing 1x 2x slower on a 500GB table where liquid clustering is enabled in the day time period column compared to partition and non partition

Barny Self · Answer

< Venkat Hari> I think your table is too small too benefit from clustering with such a low cardinality on your partitioning columns partitioning is still prob your best bet here Still working with the product team on this to investigate a few options

Venkat Hari · Answer

Thanks < Barny Self> for the answer

Venkat Hari · Answer

Still the table is less than 1TB without partition providing better I O in Databricks on AWS It could be nice to mention in Databricks doc that liquid clustering is for only very large table scenarios

Barny Self · Answer

It is not just size but cardinality as well just working on what extra info is needed to be provided in the docs as I agree with you

Venkat Hari · Answer

Ah I missed that point about cardinality you mentioned in your previous message interesting going to try on high cardinal column of 1TB table and will update it here

Oliver Angelil · Answer

gt Barny Self any updates on this My team and I are seeing read operation is performing 1x 2x slower on a 500GB table where liquid clustering is enabled in the day time period column compared to partition and non partition < Venkat Hari> what do you mean by day time period column TimeStamp Isn t that high cardinality

Venkat Hari · Answer

< Oliver Angelil> it s not that high as I will have 365 days of partition per year with 1GB of data

Oliver Angelil · Answer

< Venkat Hari> why don t you use Ingestion Time Clustering Are you sure Liquid Clustering is suitable for your use case