Is it possible to just optimize files between commit 0 and s

Question

Is it possible to just optimize files between commit 0 and some commit version N During my testing when loading table version N and then running optimize it looks like it just overwrites commit N+1 instead of trying to commit N+1 and then advancing to the next available version Sounds like a bug Or is this not possible to do

Cole MacKenzie · Answer

So if N+1 already exists say adds a some files the overwrite then leaves those files dangling and not attached to the table with no record of where they should go

Cole MacKenzie · Answer

<https docs delta io latest concurrency control html write conflicts> Looking at the table here I looks like `INSERT` and `COMPACTION` should be able to run concurrently without corruption one another

Robert · Answer

this should be possible although I am not sure that we have commit conflict resolution already included in all code path DO you have an example where taht happens

Cole MacKenzie · Answer

Thanks Robert I was looking into it more and I think it might be because I am using `AWS ALLOW UNSAFE RENAME`