Can transaction log files be safely removed from a delta table? https://github.com/delta-io/delta/blob/master/PROTOCOL.md#last-checkpoint-file

The Delta transaction log will often contain many (e.g. 10,000+) files.

while I would definitely not recommend deleting them manually, it should technically work. But you will end up with orphaned data files and should run a

VACUUM

afterwards to free up the space of the data files also. but in your case I would rather investigate why the logs are not cleaned up automatically (assuming they are older than your

delta.logRetentionDuration

and a checkpoint was recently created) OR why you do have so many logs at all. It could very well be that you have a streaming job that writes into your delta table continuously thereby producing a lot of log files

I’m actually only wondering why this part of the process is not a bit more transparent in delta. Is there any open source corresponding operation to this Databricks functionality?

I am quite sure this part is not proprietary to Databricks but is also in the OSS version. What Databricks functionality are you referring to?

I’m referring to the process used to take care of removing outdated transaction log files based on

delta.logRetentionDuration

setting

that should just work the very same way in OSS as it does in Databricks

Can you give me a bit more details? So there is in delta lake oss a routine used to remove transaction log files when they are older than the specified

delta.logRetentionDuration

period?

yes, otherwise all implementations/tools that use Delta OSS would never cleanup log entries

I did a test by adding a few hundreds of transaction log files on Databricks, (i did insert, delete, vacuum, optimize) and have

delta.logRetentionDuration

set to

2 minutes

, but no file has been removed. Moreover there is no reference in the protocol that the transaction log files should be removed or how they are supposed to be removed.

can you try to set it to

interval 2 minutes

not sure though if minute intervals actually work

Gerhard, thank you for your patience. I will try setting to

interval 0 hours

to see what happens

https://docs.delta.io/latest/delta-batch.html#data-retention (but I guess you came across this already)

did you see the functionality in action? does the cleanup happen right after creating the checkpoint or is it a managed service which asynchronously sweeps dangling transaction log entries from all the tables of the customer?

I dont know to be honest but again, if it did not work, a lot of people would have already complained I guess 😄

It is not a managed service.

thank you Dominique

The code logs to Info level the date is expiring to and how many files it deleted

i am looking at the code and see that rightafter the checkpoint gets created, the cleanup should be performed

Because it truncates the retention parameter to day, so if you are setting a short one, it won't delete until the first checkpoint on the following day