I learned something interested about the
Delta Lake transaction log protocol the other day. (
@Ryan Johnson - please confirm this description is accurate. )
The minimum reader/writer versions to consider something a “Delta Lake implementation” are reader protocol v1/writer protocol v2.
The old protocol system was a bit too strict, so Delta Lake has now transitioned to table features. Delta Lake implementations can now skip directly from reader_v1/writer_v2 to reader_v3/writer_v7.
Unlike the previous protocol versions, reader_v3/writer_v7 don’t require cumulative support for all previous table features. Once you are at reader_v3/writer_v7, you can cherry pick the table features you’ll support.
It’d be good to see how we can get delta-rs to reader_v3/writer_v7. Seems like we’re close.