Anyone know how to read a Delta table into a DataFusion DataFrame with the Python API? Here’s what I tried:

from datafusion import SessionContext
from deltalake import DeltaTable

table = DeltaTable(f"{pathlib.Path.home()}/data/delta/G1_1e8_1e2_0_0")
ctx = SessionContext()
ctx.create_dataframe(table.to_pyarrow_dataset())

Here’s the error I got:

TypeError: argument 'partitions': 'FileSystemDataset' object cannot be converted to 'PyList'

More info in the thread…

Have you tried

register_table(name,table)

?

from datafusion import SessionContext
from deltalake import DeltaTable

table = DeltaTable(f"{pathlib.Path.home()}/data/delta/G1_1e8_1e2_0_0")
ctx = SessionContext()
ctx.register_table('my_table', table)

Thanks @Jordan Fox. This worked:

ctx.register_dataset("my_dataset", table.to_pyarrow_dataset())
ctx.sql("select * from my_dataset where v2 > 5")

This didn’t work:

ctx.register_table("my_table", <http://table.to|table.to>_pyarrow_table())

This didn’t work either:

ctx.create_dataframe(<http://table.to|table.to>_pyarrow_table().to_batches())

Here’s the notebook if you’d like to take a look: https://github.com/delta-io/delta-examples/blob/master/notebooks/python-deltalake/datafusion-read-delta.ipynb But one solution is working, so this is great!

Interesting that to_pyarrow_table().to_batches() doesn't work. It's how I started working with delta/duckdb/datafusion together. https://duckdb.org/docs/guides/python/datafusion.html

@Jordan Fox - this works:

[<http://table.to|table.to>_pyarrow_table().to_batches()]

. Thank you!!