I was extra caffeinated nerd face this afternoon so as promi

Question

I was extra caffeinated nerd face this afternoon so as promised here is a document detailing how PyArrow datasets work with deltalake cc< Matthew Powers>

Will Jones · Answer

Also realized that while DuckDB and DataFusion support predicate pushdown into PyArrow datasets Polars doesn t yet I <https github com pola rs polars issues 7750 issuecomment 1541013054|commented on an issue in Polars to prod them on this>

Matthew Powers · Answer

< Will Jones> this is awesome content Well done

Matthew Powers · Answer

Where would you like to get this published Can I help at all

Will Jones · Answer

I think I ll probably split into two parts 1 the what are PyArrow Datasets and how can query engines integrate through PyArrow datasets should become a new page on our python deltalake documentation 2 The rest I might make into a technical blog post on my personal blog since it s mostly interesting implementation details which could change in the future

Matthew Powers · Answer

I like that plan