https://delta.io logo
c

Christian Pfarr

01/30/2023, 7:46 PM
Hello everyone, im experimenting with deltalake and found a little problem when querying the delta table with apache drill. Drill does not interpret INT96 data as a timestamp by default, so i had to change this drill setting to read deltalake timestamps properly (not a big deal). I´ve tried to set
spark.sql.parquet.int96AsTimestamp
to false, so i wouldnt have to set this drill setting, but it seems that the delta-table writes the data always as INT96. Is it possible to configure this via a special delta config or should this setting work and i did something wrong while setting up my spark session?
ahh... found it by myself 😄 setting
spark.sql.parquet.outputTimestampType
to
TIMESTAMP_MILLIS
does the trick for writing and drill works without special config
4 Views