@Robert the support for unity catalog (https://github.com/delta-io/delta-rs/pull/1338) seems to be languishing a bit, that draft now has some conflicts unfortunately. Do you need some help bringing that support across the line? I can deploy some Databricks resources in AWS to validate and test if necessary

I wanted to optimize a few things, specifically we can avoid a bunch of requests by just listing tables from the table summaries query. On top of that I was also contemplating some more improvements in how we create the tables, but that could be a follow up as well, as it may require some internal changes in the core. Specifically, if we lazy-load the log, as @Will Jones suggested, we can probably load the metadata from the catalog, without requiring touching the log at all .. Lastly - and this I am maybe most excited about 😄 - unity catalog also communicates credentials. As a bonus, once we can integrate object_store 0.6, we can likely get rid of some of the client code...

I thought about spinning up a PR to incorporate object_store 0.6 but stopped because of the AWS_PROFILE changes. Should I go ahead and just got that going?

yeah, we'll have to address this! I thing we are tied to 0.5* right now though until datafusion upgraded...

ah you're right, I'm so used to datafusion and arrow coupling that I didn't realize datafusion also specifies a strict dependency with object_store

well, in this case the data in the log would definately take precedence if there is aconflict, but I think the data you get from the catalog is not fetched from the log when you request it there. And we still have to scan the entire log for metadata actions, so initial load of a table could be much cheaper, for us especially also in terms on memory consumed... in the end you are right though, if you want to do more then inspepct the metadata, we'll have to load the log anyhow.