@Will Jones - here’s a notebook that shows how to query a Delta table with DuckDB. From what I recall, there is a bad way of running this query and an efficient way of running the query. I’d like to add the bad example & good example in this notebook. Can you send me some pseudocode of the bad/good, so I can understand this better?

It would be something like Bad

table = dt.to_pyarrow_table()
quack = duckdb.arrow(table)
quack.filter("id1 = 'id016' and v2 > 10")

Better

table = dt.to_pyarrow_table(filter=(ds.field("id1") == 'id016') & (ds.field('v2') > 10))
quack = duckdb.arrow(table)
quack

Best

dataset = dt.to_pyarrow_dataset()
quack = duckdb.arrow(dataset)
quack.filter("id1 = 'id016' and v2 > 10")