FYI

Question

FYI <https github com delta io delta rs pull 1226|this pull request> is a pre requisite for the next datafusion and arrow upgrades since the old `Decoder` is removed I have a follow up change to make both those upgrades possible

rtyler · Answer

Thanks for the feedback < Will Jones> I ll clean this pull request up a little bit and then I m hoping to close the week with another release for the rust bindings if everybody is on board

rtyler · Answer

So `Decoder` was deprecated and then replaced with a different API in 38 that isn t deprecated anymore arrow is wacky y all

Will Jones · Answer

Yeah I wish they had put in some guidance in the release notes on which API to use instead

Will Jones · Answer

Otherwise just have to follow the breadcrumbs in the PR and issue comments

rtyler · Answer

heh indeed

rtyler · Answer

breadcrumbs followed though I think we re in good shape now

rtyler · Answer

< Will Jones> for the checkpoints comment I might be missing the documentation but I cannot see how `Decoder` takes a defined `batch size`

rtyler · Answer

oh guffaw ReaderBuilder can build a decoder and a reader

Will Jones · Answer

`jsons` is an iterator right so you could use `Itertools chunks ` on it do handle batch size maybe Unless you just found a better way

rtyler · Answer

no I didn t I was just having the same thoughts I have no idea what `with batch size` would do with `serialize` since it s not clear what would happen if `jsons` still has elements but the batch size has been met I think chunking `jsons` is a reasonable approach to keep these batches smallish

rtyler · Answer

In the breadcrumbs you were reading did you see anything that could indeicate we can just call Decoder flush in a loop

Will Jones · Answer

Not sure I d have to look deeper into the API

rtyler · Answer

< Will Jones> would you mind