do streaming jobs and the optimize command which compacts small files conflict with each other in delta lake? i heard that they both can run into concurrency issues when started in parallel
Tom van Bussel
09/04/2023, 7:35 AM
This depends on what the streaming job is doing. If it's a pure append-only stream then there should not be any conflicts (when using the WriteSerializable isolation level). If it's a foreachBatch stream that runs a Merge statement for every batch then you are probably going to see conflicts.