This might be a dumb question - but for DataFusion to run a query against a parquet/csv file, does it have to load entire file into memory? And how does it query against a full delta table spread across multiple files?

but for DataFusion to run a query against a parquet/csv file, does it have to load entire file into memory?

No, it will try to only load the parts of the files it needs to. There's a good blog post diving into how it can do this for Parquet: https://arrow.apache.org/blog/2022/12/26/querying-parquet-with-millisecond-latency/

thank you very much