Hi All. Delta Sharing has a python client library. Is it compatible with SQLAlchemy?
My goal is to use Superset to access Delta tables using Delta Sharing.
09/13/2023, 1:59 PM
AFAIK Delta-Sharing only exposes information and give access to data files that construct a table. There's some support for e.g. predicate pushdowns but it doesn't provide any sort of query/SQL API that SQLAlchemy could use.
I assume it's possible to read a delta-sharing table into a Pandas dataframe and then use something like DuckDB to turn it into something that is SQL query-able, and wrapping that with a custom SQLAlchemy dialect 🤯
Not quite a trivial solution I guess 🙂
09/13/2023, 2:06 PM
Thanks @Oz Katz. It’s unfortunate that Delta Sharing doesn’t support SQL API because it can provide more access to other tools.
Yeah, I read a similar suggestion on using DuckDB. But as you have said, it’s a “messy”, and it doesn’t exploits many benefits of Delta features such as data skipping and predicate push down. So I doubt it will have a good performance.