We see a phenomena where queries that search only for MAX TI

Question

We see a phenomena where queries that search only for MAX TIMESTAMP in delta tables are yielding a scan and not a metadata lookup Has anyone encountered this

JosephK (exDatabricks) · Answer

This is a known issue that has to deal w how timestamps are saved There is currently an open ticket to resolve it

Sherlock Beard · Answer

Oh is it not related to min max optimization sweat smile

JosephK (exDatabricks) · Answer

I m pre coffee so I don t know the exact details but I think it s because a timestamp can be saved in either micro or milliseconds and depending on which one it will mess up the max

JosephK (exDatabricks) · Answer

Looks like min max only works with numeric and date columns for metadata queries File skipping will work on all columns

Sherlock Beard · Answer

Just to be clear we are talking about Delta oss right

JosephK (exDatabricks) · Answer

So technically we re talking about the query engine reading the files Delta is just a file format protocol and what happens with it is independent of that

JosephK (exDatabricks) · Answer

You ll get different results from the query depending on what you use to read the table In this specific example you might get better results from a traditional database such as snowflake over using databricks for that query

Yuval Itzchakov · Answer

< JosephK exDatabricks > Do you have a link to the open ticket

Yuval Itzchakov · Answer

I also understand that this optimization should only work for partition columns Not generally for any column on the table