@Robert@Will Jones (and other contributors) I had the opportunity to talk with a 🧱 VP of Product today and used the opportunity to press them for more investment in making Unity catalog useful/adopted by the open source ecosystem including Delta Lake connectors which are not directly maintained by Databricks.
I have a follow up meeting with some more product people in the next week or two The experimental support was added in https://github.com/delta-io/delta-rs/pull/1331 and I'm hoping to convince them to invest in pulling that out of experimental support and making it supported for Python and Rust
07/22/2023, 4:05 AM
As it turns out we have much more extensive Unity catalog support planned and also in preparation. Specifically focussing on dcatafusion support and eventually leverging more metada from the catalog.
THere is a working variant in a PR, but I am not happy yet how the catalog is traversed - know a solution though 🙂
07/22/2023, 4:48 PM
@Robert - that sounds awesome. Could you send me a high-level recap that I can pass on to the DataFusion community?
07/22/2023, 7:36 PM
Is this related to the unity catalog as hive metastore for external systems private preview, or some other functionality?
07/26/2023, 7:51 PM
@Matthew Powers - sure, what to recap exactly though? 🙂 - the state of the unity catalog integration for datafusion?
Its actually quite far along, right now we only have azure and PAT support though. However @rtyler mentioned he might be able to validate AWS? We essentially hoisted the auth from object store, so hopefully things work ...
07/26/2023, 8:16 PM
@Robert - yea, the state of the unity catalog integration for DataFusion would be a great little recap for the DataFusion community. I am sure they would be really interested to hear!