do streaming jobs and the optimize command which compacts small files conflict with each other in delta lake? i heard that they both can run into concurrency issues when started in parallel

This depends on what the streaming job is doing. If it's a pure append-only stream then there should not be any conflicts (when using the WriteSerializable isolation level). If it's a foreachBatch stream that runs a Merge statement for every batch then you are probably going to see conflicts.

Thanks!