Hi, is there a way to define a lifecycle policy on a delta lake table according to a partition column? Something like auto deletion of all the data older than X days according to a partition column?

assuming you're using S3, you can set lifecycle policies based on the prefix of an object, e.g. table/day=Sunday/part-000.parquet using "table/day=Sunday" in the prefix If you're talking about doing something dependent on the partition value, that might be possible but I'm not sure offhand if/how that can be implemented

But how can we rely on S3 retention without making the delta log aware of these changes? Sounds like this is going to break the table

a great deal of cleverness is required 😆 You can use S3 Object Event Notifications to look for those Delete operations and have a Lambda make the corresponding transaction log removals in the

_delta_log/

🤔 this might be a good blog post for me to write 🤔

Is this something that is supported by delta lake? How would you manipulate the delta log?

Bear in mind I am a contributor to #delta-rs so I might be a bit more comfortable than most directly working with the transaction log 😉 Anyways, the transaction log doesn't know anything other than what you tell it, it's just JSON sitting in storage. My suggestion is to come along after an object has been deleted and add the corresponding "remove" transaction. This is relatively straightforward from the Rust libraries but I'm not sure what is feasible from the Spark layer unfortunately