For some more Delta related context on

Question

< Will Jones> For some more Delta related context on I basically cannot run our delta rs optimize on Spark created parquet files with a `primitive timestamp ` column in the schema because the ParquetReader gives `Timestamp nanosecond ` from the int96 column but the Arrow schema delta rs thinks it should be using is `Timestamp microsecond ` which I believe to be the correct interpretation here My gut feel is that this will be the case for parquet files written by Delta Spark but I m not sure where the best place to introduce the necessary conversion will be Putting it in our writer feels correct but I think we would have other readers not doing the right thing on this type either

Will Jones · Answer

Why put the conversion in the writer and not in the reader

Will Jones · Answer

The one thing I m not sure of does Spark and other engines care if we write back files with microsecond timestamps if there already exist files with int96 nanosecond timestamps

rtyler · Answer

In the case of optimize the reader is just the reader from the Parquet crate itself I don t know how we d appropriately convert there since this seems like a delta problem

rtyler · Answer

I was <https delta users slack com archives C03FVMHT93Q p1681362369066279|hoping to get a pointer> to where the parquet write happens in the delta spark implementation since I wasn t able to find it last night

Will Jones · Answer

I think on the reader side you could cast each record batch as it gets read

rtyler · Answer

At least for optimize specifically that would be <https github com delta io delta rs blob main rust src operations optimize rs L306|here> but I think this would be a generalized problem no

rtyler · Answer

A colleague pointed me to <https github com delta io delta rs issues 441> which is also in the same vein which you also commented on laughing

Will Jones · Answer

Yes the general problem is why I created this issue <https github com apache arrow datafusion issues 5950>

Will Jones · Answer

IMO PyArrow gets this right you can pass the Arrow schema you want and the reader will do it s best to read into cast to that schema

rtyler · Answer

What do you think about mimicking Spark s functionality and just writing INT96 by default for timestamp unamused

Will Jones · Answer

Do newer versions still do that I thought they had deprecate and gotten rid of it in newer versions

Will Jones · Answer

or maybe I am mixing that up

Will Jones · Answer

gt Since Spark 3 0 parquet logical type `TIMESTAMP MICROS` is used by default while saving `TIMESTAMP` columns In Spark version 2 4 and earlier `TIMESTAMP` columns are saved as `INT96` in parquet files To set `INT96` to `spark sql parquet outputTimestampType` restores the previous behavior gt

Will Jones · Answer

Will Jones · Answer

IMO writing int96 should be unnecessary I think it s the read that we need to fix right

Will Jones · Answer

unless Spark can t read tables that are a mix of the two timestamp types I haven t tested that

rtyler · Answer

unfortunately this parquet file was written by Spark 3 2 last week face palm I need to figure that out slightly frowning face

Robert · Answer

I would agree that passing a schema to the reader and the reader trying to cast all batches to that schema seems like the most desirable way Particularly within an operation the only thing we really have full control of is the read path since we cannot control all writers right

rtyler · Answer

The reader in this case is the so if we wanted to address this we would have to be modifying that API or providing an alternative With RecordBatch we can allegedly override the schema but I m not sure just yet what that might do with this data set thinking face

Will Jones · Answer

I don t think `with schema` will work it looks like it only handles metadata and will fail if data types change I think we ll need to write our own function that casts record batches by traversing the schema and applying functions from <https docs rs arrow cast 37 0 0 arrow cast cast fn cast html>

Will Jones · Answer

IMO we should submit that function upstream later if we can

rtyler · Answer

heh I was just looking at that <https github com apache arrow rs pull 4028|It was introduced> in arrow 37 too which would mean I d have to finish our upgrade to use it anyways What hell laughing