Matthew Powers

06/16/2023, 11:47 AM
I learned something interested about the Delta Lake transaction log protocol the other day. (@Ryan Johnson - please confirm this description is accurate. ) The minimum reader/writer versions to consider something a “Delta Lake implementation” are reader protocol v1/writer protocol v2. The old protocol system was a bit too strict, so Delta Lake has now transitioned to table features. Delta Lake implementations can now skip directly from reader_v1/writer_v2 to reader_v3/writer_v7. Unlike the previous protocol versions, reader_v3/writer_v7 don’t require cumulative support for all previous table features. Once you are at reader_v3/writer_v7, you can cherry pick the table features you’ll support. It’d be good to see how we can get delta-rs to reader_v3/writer_v7. Seems like we’re close.


06/16/2023, 3:40 PM
right after gets merged, I have some work lined up to get us to writer V2. After that I guess it makes sense, to go for V3 / V7, since as you mentioned the intermediate steps can be skipped. Then we still would have to implement all the features, but can do it one by one 🙂
👍 1

Matthew Powers

06/16/2023, 3:50 PM
Awesome, great progress!

Ryan Johnson

06/30/2023, 8:57 PM
we still would have to implement all the features, but can do it one by one 🙂
That's exactly the goal of table features, yes -- break the horrible cumulative dependency wall
(sorry, a bit late to this thread -- I'm not very good at navigating slack notifications across multipole workspaces)