https://delta.io logo
m

Michael Nacey

05/25/2023, 3:22 PM
Hey guys: loving using delta-rs in python to break away from spark for the more simple things I want to do. One question, how does one set the delta option (for write) of "dataChange = false"? I have not found a way to do this. Thanks!
w

Will Jones

05/25/2023, 3:27 PM
That's something that's implicit in the operation you are doing, so we do that for you. What's the use case where you want to set it? Is there somewhere you think we are setting it incorrectly?
n

Nick Karpov

05/25/2023, 5:05 PM
interesting - it'd be nice to expose this as a user option where appropriate so users have the flexibility to BYO-compaction/data layout, it's possible to pass it in spark as a dataframe writer option (https://docs.delta.io/latest/best-practices.html#-delta-compact-files)
👍 2
m

Michael Nacey

05/25/2023, 6:39 PM
As Nick mentioned, it's for BYO compaction purposes. We do some de-duplication as part of our compaction routines.