I was trying to read and write data from a pyarrow dataset to delta lake using polars but ran into some data type issues with categorical columns and timestamps. my data was originally written with hard-coded pyarrow schemas and we used dictionary types pretty heavily. is this still supported in delta lake?

pa.schema(
        [
            ("load_timestamp", pa.timestamp("ns", "UTC")),

            ("survey_completed", pa.timestamp("us", "America/New_York")),
        
            ("business", pa.dictionary(pa.int8(), pa.string())),

and the timestamp error:

Exception: Schema error: Invalid data type for Delta Lake: Timestamp(Microsecond, Some("America/New_York"))

We haven't yet implement automatic casting of data types, so you have to cast to the exact data types supported by Delta Lake. For timestamps, we only support no time zone or UTC at the moment.

got it, thanks

Yes, I think so