Hello I have a delta log folder delta log having millions of

Question

Hello I have a delta log folder delta log having millions of files It is retaining all old log files Is there a way to remove archive consolidate the delta log log files

Scott Sandre (Delta Lake) · Answer

Scott Sandre (Delta Lake) · Answer

gt `delta logRetentionDuration = interval lt interval gt ` controls how long the history for a table is kept The default is `interval 30 days`

Scott Sandre (Delta Lake) · Answer

we keep the delta logs for a default of 30 days

Ashutosh Pandey · Answer

Thanks for a quick response

Scott Sandre (Delta Lake) · Answer

slightly smiling face

Ashutosh Pandey · Answer

Where to set this as env variable

Ashutosh Pandey · Answer

cluster level

Scott Sandre (Delta Lake) · Answer

this is a table property

Scott Sandre (Delta Lake) · Answer

Ashutosh Pandey · Answer

thanks