https://delta.io logo
r

Roel Knitel

03/21/2023, 6:40 AM
Is it possible to retain certain points in time using Vacuum command? For example always retain the data for the first of the month.
g

Gerhard Brueckl

03/21/2023, 7:57 AM
currently no - I guess the way to solve this would be to create a second snapshot table to which you copy the data on every first of the month
r

Roel Knitel

03/21/2023, 8:32 AM
Good suggestion for a workaround! Any idea if this feature is on the roadmap?
g

Gerhard Brueckl

03/21/2023, 8:52 AM
honestly I do not know - but I also do not think it would be a good idea to store the snapshots together with the other transactions. you would end up with holes in your version history which could easily mess things up or at least be very confusing
r

Roel Knitel

03/21/2023, 9:52 AM
Thank Gerhard - I can follow your logic. On the other hand it seems a bit over the top to store 29 days of data when you only need a single day per month of each table as per requirement. For a whole year you would have to store 29 *12 = 348 days of data that you never use but have to pay for.
g

Gerhard Brueckl

03/21/2023, 10:06 AM
why 29*12 times? if you say you only store the data for the first of the month this would make 1*12 times the original data volume per year
r

Roel Knitel

03/21/2023, 10:09 AM
I would opt for a Vacuum parameter that let's you Vacuum 'selectively'. From what I have read you can only Vacuum data older then X hours. Using your workaround you could indeed store only the needed data but this seems like a lot of 'manual' config. Hope this makes sense
k

Kees Duvekot

03/23/2023, 10:41 AM
Sounds like you want to use the history of delta as some kind of "backup" mechanism .. retaining selective versions to create a "one backup at the beginning of the week for 4 weeks, then switch to monthly copies after that . and maybe even one for a yearly. But I am not sure if this delta history in concept was intended for that exact purpose .. so I would advise not try to use the history for that purpose.
r

Roel Knitel

03/23/2023, 5:03 PM
Hi Kees, thanks for your response. It would not only be for backup but also for analysis and auditing purposes. It's true that it would follow more traditional backup patterns in terms of frequency etc. Regardless of the original intention it seems Delta is almost there should one be able to Vacuum 'selectively'. True 'timetravel' without the necessity to store all daily runs under the hood, especially useful in Overwrite mode.
4 Views