With the object store abstractions we have a pretty reasonable way to plug new backends into delta-rs. I once upon a time hacked on supporting #delta-sharing natively, and was discouraged for a number of reasons.
I think we now have enough of an interesting abstraction layer to support sharing at the rust layer, I'm curious for feedback on supporting delta-sharing like it's Just Another Storage Backend (read-only)
05/22/2023, 1:50 AM
I'm not sure if I'm the right person to ask a question here though, I'm curious. Are you considering a feature like this?
We would have to use a URL scheme of some form to indicate that the delta-sharing storage backend should be used, but yes something like that
05/22/2023, 2:50 AM
@rtyler Thank you for your answer. I'm glad it provided me with a better understanding. Your proposal is quite interesting!
05/22/2023, 5:02 AM
From my perspective quite interesting as well. Without having thought about a design too much, it seems we are working up towards separating log storage from data storage (@Nick Karpov) and this may be another use case where this would come in handy.
Implementing a catalog on top of the sharing api seems straight forward as well.