https://delta.io logo
k

Kevin Lim

09/18/2023, 9:03 PM
Does delta-rs suport automatic schema evolution? (addition of columns/renaming columns/missing columns in newer versions of data) on append?
w

Will Jones

09/18/2023, 9:04 PM
I don't think so. I think those features would require column mapping, which we don't yet support.
k

Kevin Lim

09/18/2023, 9:09 PM
ah, thanks.
do you know if delta-rs is only maintained by volunteers?
w

Will Jones

09/18/2023, 9:16 PM
Yes, all volunteers. It is no ones day job to maintain delta-rs right now.
k

Kevin Lim

09/18/2023, 9:18 PM
Ah thanks for the info!
w

Will Jones

09/18/2023, 9:26 PM
I think there are certain companies using delta-rs in production, so they can about maintaining the particular pieces they use. But the overall maintenance and direction is volunteer driven.
šŸ‘€ 1
šŸ«£ 1
r

rtyler

09/18/2023, 10:44 PM
there are a lot of people contributing to delta-rs in a professional context (myself included) but @Will Jonesā€™s first point is spot on: it's nobody's job explicitly to maintain delta-rs šŸ˜† My employer (Scribd) created it originally but our focus has always been on the write path, there's a few folks who specialize more on the read path, some with Rust, some with Python šŸ™‚ We're a motley crew that's for sure
s

Simon Thelin

09/19/2023, 3:14 PM
We use delta-rs in production, for automatic schema evolution you have to currently read in the whole table, align with the change and overwrite, it works for small tables but for really large on it is a pain atm. Right now we use spark for large tables so it works out okay. We also combine trino and delta-rs together in tandem and handle merges via trino and load in memory tmp delta tables via delta-rs to handle those scenarios atm.