Is it theoretically possible that the transaction log gets updated faster than it's being read, resulting in the table being loaded for eternity? It seems to me that there's no way of knowing the max_version when starting to load the table. Tangentially related to https://github.com/delta-io/delta-rs/pull/1549 because there will necessarily be some unnecessary requests when doing concurrent reads to the log - which would not be the case if max_version was known

Yeah in theory the current implementation can get stuck in loop fetching the next version if write speeds exceeds read speeds. The solution you identified will remove that infinite loop. In the object store list_with_offset maps the the S3 param start-after that you mentioned. https://github.com/apache/arrow-rs/blob/d9719d1f2d21ed0c75ffd5c8e8c4fb15ee5ebe54/object_store/src/aws/client.rs#L586-L588

Yeah I noticed that, and added a quick and dirty implementation here in the PR: https://github.com/delta-io/delta-rs/pull/1549/files/b1873745a310117d967d435f229bd77780762e29#diff-3b5c80af9e5527fbeaa6[…]0ba0471c58f243a29dfa99R553 Unit tests run OK but haven't tried it against S3

Neat. We have macros to help with integrations tests against the various objects stores: See: https://github.com/delta-io/delta-rs/blob/main/rust/tests/integration_checkpoint.rs#L21 For benchmarking this repo has

criterion

included which is really great for that work