Hi All, what's the impact of the delta kernel going to be on the delta-rs project?

We are collaborating to work on the Rust implementation of delta-kernel. once ready, delta-kernel will replace much of the low-level code in delta-rs as part of a refactor. One of the biggest changes we anticipate is that we will no longer eagerly load all table metadata and statistics, making it feasible to operate on very large tables.

And with regards to the planned ADBC api? Is that going to be part of the kernel

No that is a higher-level wrapper

Chiming in as a developer actively working on the kernel implementation. Realistically speaking it will not change much for delta-rs except that it will make it easier for us to adopt newer protocol features. delta-rs has grown (🥳) quite a lot but rather organically over the past couple years and we have some API layer improvements that will be enabled by kernel. As @Will Jones mentioned there are some performance benefits that we'll be able to gain, but not necessarily because of any massive shift in how delta-rs operates but rather because we' have learned a lot about where things are slow and inefficient in delta-rs when dealing with the commit log, and we have an opportunity to provide cleaner streaming interfaces with kernel

how about delta uniform? will the release of delta universal format (uniform) reflect on delta-rs? It’s somehow confused here. Correct me if I am wrong but delta-rs will also follow the uniform, right?

There are some updated details in the procotol document: https://github.com/delta-io/delta/blob/master/PROTOCOL.md#iceberg-compatibility-v1 We've not yet discussed how/why that might come into delta-rs just yet, but the assumption should be that kernel supports these new features and delta-rs would get them "for free"

I am curious, when do you all expect that delta-rs (incl. Python bindings) will be on par with delta-spark in terms of features? I have to develop a roadmap for our team and see when we could possibly make the switch to delta-rs 🦀 🙂

what features ya need @Ion? 🙂

Kernel will bring us closer, but thats really hard to say in general. The best way to make that faster is proving new contributors. If there’s specific features you need, you might consider contributing them. We’d be happy to provide guidance and PR reviews.

@rtyler mostly CDC read and write support, and Merge :). For simple append only or overwrite use cases I already started using it, but for complex stuff I still have to go through spark @Will Jones I have 0 experience with Rust, so I would first need to get a general understanding of the language ^^

oh thanks, very excited for the upcomming delta-kernel in Rust