Hi Team Delta, I want to understand, on what happens when we perform Vacuum on a Delta table, as per below scenario ❓❓❓❓🤔🤔🤔🤔 Scenario - I have a Delta Table of 1TB with only 20GB as active and remaining as inactive and current cluster with 1 Driver and 2 Executor (Executor having 8GB memory each and 4 cores each) I want to perform Vacuum once a day but this should not impact my Job processing performance. What shall I do? Shall i Increase the cluster size a or Its is not required?

You shouldn't set up a cluster this way. It's better to scale up a single node rather than have 2 tiny nodes.

Hi @JosephK (exDatabricks) Spinning up a new cluster for vacuum is not possible, we have to perform it on the same cluster, That is why want to confirm if I can allocate a dedicated thread to this process without scaling up the cluster, and it it won't impact my current processing

My comment had nothing to do with vacuuming. You should never set up a cluster that way for any process.

If I understood correctly, you simply want to vacuum a Delta table which is also constantly written to. In that case you can simply run VACUUM from another cluster and it will have no impact on your processing performance.

Why do you have the constraint it has to be run on the same cluster?

Don't have permission to spin up another cluster 😄