https://delta.io logo
l

Lennart Skogmo

02/05/2023, 9:33 AM
Does anyone have any tips about the best way to atomically replace the content of a delta table? With other tech you have some options like, transaction truncate & insert, insert into new table and swap tables with atomic rename, and ofc merge into if we have keys.
m

Maximilian Jackson

02/05/2023, 11:09 AM
delte, update and upsert with merge examples. https://docs.delta.io/0.4.0/delta-update.html
👍 1
l

Lennart Skogmo

02/05/2023, 11:35 AM
Seems like this might do the trick as well. I found a Databricks forum post claiming this operation is also atomic. So there should be no possibility of encountering a missing or empty table if someone tries to read at the same time. Also I guess it could be something unique to Databricks.
df.write.mode("overwrite").format("delta").saveAsTable("tablename")
m

Maximilian Jackson

02/05/2023, 1:09 PM
chances are low + i try to give my users only Views to consume
l

Lennart Skogmo

02/05/2023, 1:23 PM
Did a quick test now. It got recorded as a 'CREATE OR REPLACE TABLE AS SELECT' operation. Also it did not break a view I had pointing to the table so then all seems good as long as care is taken not to mess up the schema.
3 Views