do you know if delta-rs is only maintained by volunteers?
09/18/2023, 9:16 PM
Yes, all volunteers. It is no ones day job to maintain delta-rs right now.
09/18/2023, 9:18 PM
Ah thanks for the info!
09/18/2023, 9:26 PM
I think there are certain companies using delta-rs in production, so they can about maintaining the particular pieces they use. But the overall maintenance and direction is volunteer driven.
09/18/2023, 10:44 PM
there are a lot of people contributing to delta-rs in a professional context (myself included) but @Will Jones’s first point is spot on: it's nobody's job explicitly to maintain delta-rs 😆
My employer (Scribd) created it originally but our focus has always been on the write path, there's a few folks who specialize more on the read path, some with Rust, some with Python 🙂
We're a motley crew that's for sure
09/19/2023, 3:14 PM
We use delta-rs in production, for automatic schema evolution you have to currently read in the whole table, align with the change and overwrite, it works for small tables but for really large on it is a pain atm. Right now we use spark for large tables so it works out okay.
We also combine trino and delta-rs together in tandem and handle merges via trino and load in memory tmp delta tables via delta-rs to handle those scenarios atm.