Found an issue where the column statistics fail to parse dur

Question

Found an issue where the column statistics fail to parse during query time for `timestamp` fields <https github com delta io delta rs issues 1372>

Cole MacKenzie · Answer

I am seeing this in `0 10` Haven t tried `0 11` yet but I think this was a change in arrow

rtyler · Answer

I wonder if this might be the same issue as <https github com delta io delta rs issues 1286> showing up in a different spot

Cole MacKenzie · Answer

Might be something going on in here <https github com delta io delta rs blob main rust src action parquet read mod rs L247>

Cole MacKenzie · Answer

It might be I spot checked a few ` json` files in the ` delta log` and they look to be proper RFC3339

Cole MacKenzie · Answer

Looking at the ` checkpoint parquet` files I see them as ``` Tag name=EdgeEndTimestamp type=INT64 convertedtype=TIMESTAMP MICROS repetitiontype=OPTIONAL ```

rtyler · Answer

parquet things timestamps should be one thing Delta likes to pretend timestamps are INT96 s which is super deprecated

Cole MacKenzie · Answer

Yeah I cry every time I see `INT96` anywhere

Cole MacKenzie · Answer

Cole MacKenzie · Answer

Is that just missing `TimestampMicros` block

Cole MacKenzie · Answer

Cole MacKenzie · Answer

tada

Cole MacKenzie · Answer

I hope

rtyler · Answer

does this work for your use case with a real delta table already

Cole MacKenzie · Answer

Copying the table over to try now If the encoding in parquet is `type=INT64 convertedtype=TIMESTAMP MICROS` then the match statement should have that type right

Cole MacKenzie · Answer

These parquet files were generated by delta rs v0 10

rtyler · Answer

it might I m not sure tbh