Hey there, I’m facing an interesting problem where my end-users will often rewrite a partition with the same data multiple times per day. The problem is that the transaction log will get “flooded” with writes while having the same data in the table. End-users are using notebooks, one code cell writes the data and will get often executed because of rerunning the whole notebook. I wonder if there is a “smart” way to avoid writing data when it’s the same beside comparing the data row by row (which sound a bit complex)… Let me know if anyone has any ideas or tips, thank you!

Check idempotent writes feature, I have not tried it for writes on single partitions but should work fine. If you can find a valid combination of appId and txnVersion, that would prevent multiple identical writes. https://docs.databricks.com/delta/idempotent-writes.html https://towardsdatascience.com/idempotent-writes-to-delta-lake-tables-96f49addd4aa

Awesome, thanks!

can

MERGE

be used instead? partition overwrites come from a time where you couldn't do row-level changes - and now you can.

Merge can be used but it will create a new version in the transaction log, I want to avoid that if the data is the same. I don’t want to see a lot of identical versions when describing the history of a table.

it's true,

MERGE

without any changes to the data will still generate a new transaction. would it be satisfactory if you could filter out these non-changing transactions from the history?

Yeah they can be filtered out when reading the history based on operationMetrics

i think MERGE is better then? you can just filter out the "dud" commits. if you do a partition overwrite, it's harder to differentiate commits that actually did something with commits that just overwrote but didn't change anything