Abolfazl karimian
07/11/2023, 7:56 AMdelta_table = DeltaTable.forPath("path")
delta_table.*logRetentionDuration("interval 1 days")*
delta_table.deletedFileRetentionDuration("interval 4 days")
session.sql("vacuum delta.path")
these configurations do not work at all and won't delete files from disk until their default value is met!Martin
07/11/2023, 5:44 PMdelta.deletedFileRetentionDuration
and delta.logRetentionDuration
are _*Table properties*_:
https://docs.delta.io/latest/delta-batch.html#table-properties
example:
spark.sql(f"ALTER TABLE delta.`{path}` SET TBLPROPERTIES ('delta.logRetentionDuration' = 'interval 4 days', 'delta.logRetentionDuration ' = 'interval 1 days')");
Abolfazl karimian
07/19/2023, 11:33 AM