Hello Delta community, I want to check the progress of open source all Delta to Delta OSS (announcement). I really like Delta but the open source version is missing some critical features like auto compaction. I am also curious about why there is no merged pull request for OSS version in last month
05/17/2023, 2:14 AM
Hey Li, I think that’s a fair question and I’ll put this on us for not providing the initial roadmap that normally do at the beginning of the year.
We’ve been communicating via Slack as well as through Github issues and more recently Last Week in a Byte (go.delta.io/lastweek) about the many releases across our various APIs including connectors (Flink, Standalone, Power BI), Rust, Spark, Trino, etc. But this informs you a-priori when we should have provided the roadmap up front.
From the context of your question, the way our GitHub is setup, it’s under reporting the PRs that have been merged. If you refer to the releases themselves - such as our latest one https://github.com/delta-io/delta/releases/tag/v2.3.0 - you’ll notice many OSS contributions that have been merged. Saying this:
• We are working to fix the way our GitHub is setup so that way the PRs are reflected through the regular APIs (this is on us, not blaming GitHub at all)
• We need to provide better planning docs (like we did in the past) so everyone can review and approve the overall plan vs the piecemeal approvals that we have been doing.
Hope this helps and feel free to ping me or any of the contributors and maintainers if you have any questions. (And to those that I still owe answers, I will get to them but feel free to re-tag me).
05/17/2023, 1:38 PM
Thanks! This is very helpful to us. The regular roadmap/planning update and Github PR stats definitely will give people more confidence to onboard to Delta. Looking forward to this change and more exciting features in Delta!
05/17/2023, 2:22 PM
Completely fair - a lot of this is on me. I used to post the roadmap (only a month late) like https://github.com/delta-io/delta/issues/1307. But I kept on putting it off and now we’re in May. I’ll try to get my act together for July (well…August) ;)