@Will Jones - this is awesome content!! Well done!
Where would you like to get this published? Can I help at all?
05/10/2023, 10:21 PM
I think I’ll probably split into two parts:
1. the “what are PyArrow Datasets?” and “how can query engines integrate through PyArrow datasets?” should become a new page on our python deltalake documentation.
2. The rest I might make into a technical blog post on my personal blog, since it’s mostly interesting implementation details (which could change in the future)