@Robert @Will Jones (and other contributors) I had the opportunity to talk with a 🧱 VP of Product today and used the opportunity to press them for more investment in making Unity catalog useful/adopted by the open source ecosystem including Delta Lake connectors which are not directly maintained by Databricks. I have a follow up meeting with some more product people in the next week or two The experimental support was added in https://github.com/delta-io/delta-rs/pull/1331 and I'm hoping to convince them to invest in pulling that out of experimental support and making it supported for Python and Rust

deltalake

packages 🤞

As it turns out we have much more extensive Unity catalog support planned and also in preparation. Specifically focussing on dcatafusion support and eventually leverging more metada from the catalog.

@Robert - that sounds awesome. Could you send me a high-level recap that I can pass on to the DataFusion community?

Is this related to the unity catalog as hive metastore for external systems private preview, or some other functionality?

@Matthew Powers - sure, what to recap exactly though? 🙂 - the state of the unity catalog integration for datafusion? Its actually quite far along, right now we only have azure and PAT support though. However @rtyler mentioned he might be able to validate AWS? We essentially hoisted the auth from object store, so hopefully things work ...

@Robert - yea, the state of the unity catalog integration for DataFusion would be a great little recap for the DataFusion community. I am sure they would be really interested to hear!