Usman Zubair
08/17/2023, 10:11 PMDelta
for data arriving throughout the day. They currently use pandas
but are switching to polars
. They are encountering some challenges:
1. Polars Categorical data type is not supported in a delta format (use strings as categories to minimize data size)
2. Write a pyarrow to Delta (they are able to cast a polars table as a pyarrow table)
(1) is not supported yet so they would have to explicitly cast but is there a solution for (2)?