Hello! I've been testing the Python

deltalake

library in AWS Lambda & S3. It works nicely, but I noticed that query latency varies quite a lot depending on how many files there are in the commit log. If I read the

delta-rs

code correctly, the

update_incremental

function reads and processes all files in sequence, is that correct? And if so, would it be possible to make concurrent requests to the storage instead and buffer the results? https://github.com/delta-io/delta-rs/blob/a7389d581aceeec7187ccdfd1cbb80e7ac0747f9/rust/src/delta.rs#L472

Ideally we should be making checkpoints, and then you are reading at most 10 files. But we could also read files concurrently as well

I'm only using this setup to read, Databricks with Unity catalog is taking care of the checkpointing. And for some reason it seems to create 50 or even more files between checkpoints

Spark / Databricks allows you to configure the interval in which checkpoints are created - if that can be altered, this is a powerfuil way to reduce latency. Less effectuful, but also useful should be to disable laoding tombstones - you don't need them for read-only scenarios.

Thanks, that's good info. I couldn't find much information about checkpointing in Databricks, they just say it uses an "optimized frequency". I would love to contribute, I already played with the code a bit but ... Rust is easier to read as a complete beginner than to write 😅

we are always happy to support on any PR - giving feedback like here is also a great way to help/contribute.

Is it OK if I open a wip PR for this? I've got something that seems to work, but it could use some expert eyes

@Eero Lihavainen yes of course!

PR here https://github.com/delta-io/delta-rs/pull/1549

I saw that come through, let me convert this to a draft and we can discuss more there

Thanks!