I think once https://github.com/apache/arrow-rs/pull/4305 is released, I'm good to switch to using multipart upload when writing data files. There any other blockers for that switch that anyone knows of?

no objections from me

also none from me. I did want to have a look into writing for our operations as well, but one level up. Datafusion is moving towards doing the write in execution plans and experimenting with a

DataSink

trait. (which is kind of like our first attempts, minus the Sink 🙂) On first glance I think it does not quite fit our needs, but I wanted to experiment a bint non the less...

Are there GH issues or PRs describing this DataSink? Want to read to learn more