John Darrington

06/10/2023, 7:50 PM
Not sure if you can help with this one - looking at this in DataFusion It says it uses ObjectStore to read the files but I'm not seeing any methods for passing in an ObjectStore. Does it infer and build it based on the file uri? And if so, does that mean I'd need to include auth information in that URI somehow?

Will Jones

06/10/2023, 8:31 PM
Not an expert in DataFusion, but IIRC there’s some central place where you register them in DataFusion and then it dispatches to the right one based on the scheme and part of the path (bucket) of the path. There some place in our delta-rs code where we register our object stores


06/10/2023, 10:40 PM
Object stores are registered on the session context. The delta table itself implements the table provider trait, and handles registration of a suitable object store internally.

John Darrington

06/11/2023, 2:06 AM
Ah thanks
I guess I can just set the right env vars for object store and have it infer from the URL - that might be easiest