Hi Everyone, Good Day, Recently got a chance to implement Delta-sharing protocol specification in Python using amazing deltalake and FastAPI python packages. Thanks to detailed Protocol specification documentation combined with deltalake python package capabilities makes implementing delta-sharing protocol in python a breeze Kudos to the delta-sharing community for wonderful documentation on REST protocol specification ! Looking forward to collaborate and improve this implementation . 🙂 (Tried this delta-sharing protocol for Iceberg Table format as well - super alpha state ) Blog Post: https://guruengineering.substack.com/p/lakehouse-sharing Github: https://github.com/rajagurunath/lakehouse-sharing

Is there a good tutorial for installing Delta-Share on my local computer and using my shared tables on PBI for example?

Hello everyone - We are using Delta Sharing to enable our customers to access their data in Power BI, but our customers are complaining that the imports are taking way too long (hours) because they have no way to simply ask for e.g. the last 30 days of data or some filter like that. Are we missing something obvious? because if the user has to reimport their 100 million + rows of data every time, then this will be unusable for them. Any tips/tricks or features, capabilities we are unaware of? Thank you in advance for your insight.

Working on a Delta Sharing server implementation to enable our (InfluxData/InfluxDB) customers to access their Parquet data directly. I see that Presto and Trino are mentioned as an "Open Source Client" and part of "more are coming soon". Is there any work-in-progress that I could use to test my Delta Sharing server? The Pandas example in the delta-sharing repository is fine, until it fails to fetch the files from my development Minio server.

Hi I'm looking to integrate Delta sharing in our Data collab platform. But I would have some questions. Would someone could be available for a short chat?

Hi Team, Can delta-sharing be used for on-prem clusters as well for hdfs file systems ?

Hello everyone, can delta sharing be used inside Azure ML notebooks either via pandas or pyspark?

Hello Everyone, Do you know if there is a plan to integrate delta-sharing with Azure Active directory?

Hi All Why does delta-sharing's change data feed API does NOT include table schema changes? For example, when a column is added to a delta table and delta-sharing's reference server's change data feed API is invoked, why we do not see metadata change event in the output?

Hi everyone, I’m developing new open-source delta sharing server named deltastore. My main goals are near real-time query against the up-to-date regardless of the size of deltalogs, and access controls for data governance. Belows are key features. • Caching DeltaLogs ◦ We support deltalogs caching using a key-value storage like RocksDB. We create a cache database per each deltatable, and update newly appended deltalogs to the cache database incrementally and periodically to keep the up-to-date. ◦ We support partition filterings using range queries if you are using RocksDB for deltalogs caching. A key in a cache database is a AddFile’s path which contains partition columns and values as a prefix. So, we can scan deltalogs very fast using a prefix which is generated from predicate hints. • Configuration and Authorization ◦ We support simple REST protocols to help you develop your own configuration and authorization servers. Please let me know if you need any further information! Enjoy! Thank to all contributors for delta, delta-sharing, and delta-standalone! https://github.com/dataplatform-lab/deltastore

Is there any swagger or redoc like-ish OpenAPI documentation for the Delta Sharing Protocol? I assumed Delta Sharing Protocol is compliant wth OpenAPI. Or is it a wrong assumption?

x-posting: https://www.databricks.com/blog/using-structured-streaming-delta-sharing-unity-catalog

Hello everyone, I am trying to consume tables using Databricks delta-sharing from Qlik sense, do you know if there is any documentation that can help me with this or some way that I can do this?

Hi there, is there an easier way that following spec changes than reading the GitHub logs?

Hi all, I've been experimenting with delta-sharing a bit and I have a question regarding the scope and goals of the project. I would like to build a more "generic" data server where a client can send arbitrary JSON as a request and get a response in delta-sharing's API Response Format after some amount of computation on the server-side. Seems like, there are several design decisions around delta-sharing that make this kind of use case impossible. 1) Delta Sharing enforces somewhat sql-like (maybe database-like would be a better term) semantics for a request (share/schema/table structure and the concept of a predicate). 2) More importantly,

query

endpoint expects data response to be immediately available. While this might have made sense for the original use case, I think it's still unnecessarily limiting. Some mechanism by which a server would optionally return a token and allow the client to poll for results until they're available would free the protocol up for various use cases. What I'm probably actually looking for is a "lower-level" sister protocol similar to delta-sharing, which defines the format of the dataset metadata being returned but makes no other assumptions about the service. A good analogy would be the difference between Arrow Flight and Arrow Flight SQL from Arrow project where the latter is much more opinionated. Would you consider something like this outside of delta-sharing's scope?

Are there any plans to provide a multi arch docker container. I tried to bump the version of sbt-native-packager to the latest to try it out but end up in a dependency hell

Hi All, We are trying to read data on Power BI from delta tables stored in our systems using delta-sharing server- We are using open source delta lake to create & access delta-tables. Is it possible to create "Share" object from data provider side on non-databricks version. TIA.

Hello all, I am trying to use the load_as_spark() function on a jupyter notebook, but I keep getting this error .. I installed and imported delta_sharing I setup the credentials and created the client object .. I also tried out load_as_pandas and it worked normally I installed and imported pyspark I created the spark session any ideas what I can do ??

Py4JJavaError: An error occurred while calling o33.load.
: java.lang.ClassNotFoundException: 
Failed to find data source: deltaSharing. Please find packages at
<https://spark.apache.org/third-party-projects.html>
       
	at org.apache.spark.sql.errors.QueryExecutionErrors$.failedToFindDataSourceError(QueryExecutionErrors.scala:587)
	at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:675)
	at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSourceV2(DataSource.scala:725)
	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:207)
	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:185)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
	at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: deltaSharing.DefaultSource
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	at org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$lookupDataSource$5(DataSource.scala:661)
	at scala.util.Try$.apply(Try.scala:213)
	at org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$lookupDataSource$4(DataSource.scala:661)
	at scala.util.Failure.orElse(Try.scala:224)
	at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:661)
	... 15 more

Hi everyone, We are trying to use delta sharing on databricks, had a couple of questions on that I ) Is there any limit to the number of shares/recipients assigned per share ? II ) If I have tons of recipients and multiple shares, would this affect my jobs on data-bricks while accessing the delta tables on latency/operational basis considering delta sharing enables recipients to directly fetch from the blob store.

Hi, I'm new to delta sharing and I have a question on the license it is under. The repo states Apache License 2.0, whoever the the yaml file of the protocol description is listed as AGPL 3.0 (here: https://github.com/delta-io/delta-sharing/blob/main/delta-sharing-protocl-api-description.yml#L11). Does anyone know the thinking to that? In my scant understanding of licensing, this would not have any bearing on the project today, as the yaml file is not used to generate the APIs (at present) so the AGPL license doesn't seem to extend to the delta-sharing project overall, but seems a bit out of place to have that one file licensed differently. And someone might in future add a yaml->code generator which would very much change that.

Wasn't there a DeltaShring pluging for Excel? https://www.linkedin.com/feed/update/urn:li:activity:7011208897881378816/

Anyone with experience on Azure Databricks using delta sharing?, I have followed all the steps , enable the metastore for sharing, create a share a recipient and then install the python package pip install delta-sharing basically following this doc: https://github.com/delta-io/delta-sharing/tree/main nevertheless, all my shares are getting the error: FileNotFoundError: someone with ideas?

image.png

Hello, just one simple question. How can we join the delta-sharing ecosystem ?

Hello! I wonder If I could use a delta-sharing protocol within a private corporate hadoop system without using any cloud storage.

Hy all , i am new to delta lake and i have pipeline which saves data locally in delta tables. I want to know can i use delta share for local delta tables so that analysts can access the data or is there any other tool which can read locally save delta tables.

Hi All. Delta Sharing has a python client library. Is it compatible with SQLAlchemy? My goal is to use Superset to access Delta tables using Delta Sharing.