https://delta.io logo
i

Ion

07/18/2023, 10:36 PM
Would it make sense to have an optional pyarrow read schema for Delta-RS in python?
w

Will Jones

07/18/2023, 10:38 PM
What do you mean by "read schema"?
Could you provide an example of how it would be used?
r

Robert

07/19/2023, 5:04 AM
Elsewhere I have seen schemas used as a means to define projections and type casts as in "try to ready the table into the given schema" - is that what you mean @Ion?
👍 1
i

Ion

07/19/2023, 6:31 AM
@Robert yes that's what I mean.
So while reading a delta table with a column deltatype List, I would like to immediately cast it to a pyarrow FixedSizeList. It should work in a similar fashion as when you're writing a delta table and you provide a "schema (Optional[pyarrow.lib.Schema])" but now it would be used for reading it. @Will Jones