Hi folks I am having concurrent write errors on a delta table. The table is partitioned by day and I am using replaceWhere to filter out partitions to write. • I have a job that writes past hour only • And another job that writes past data [D-30, D-1} This way the hourly job and the daily one will never write to same partition at same time... but for some reason I am still getting the

ConcurrentAppendException

error:

Files were added to partition [event_date=2023-09-18] by a concurrent update

This log is from the daily job that have a

replaceWhere = event_date between '2023-08-18' and '2023-09-17'

and I am sure that there is no 2023-09-18 data on that dataframe. So my idea now is use upsert on hourly run and continue with replaceWhere on daily one... but I don't know if daily will continue to fail since it will be an insert and an update operation at the same time.... Any thoughts? Thank you

are you and @Lori colleagues? 😆 close timing on these questions

no haha

Alec and I asked the similar questions haha. I am feeling my spark conf does not work, but I do not know why.