Alec Ventura

09/18/2023, 9:38 PM
Hi folks I am having concurrent write errors on a delta table. The table is partitioned by day and I am using replaceWhere to filter out partitions to write. • I have a job that writes past hour only • And another job that writes past data [D-30, D-1} This way the hourly job and the daily one will never write to same partition at same time... but for some reason I am still getting the
Copy code
Files were added to partition [event_date=2023-09-18] by a concurrent update
This log is from the daily job that have a
replaceWhere = event_date between '2023-08-18' and '2023-09-17'
and I am sure that there is no 2023-09-18 data on that dataframe. So my idea now is use upsert on hourly run and continue with replaceWhere on daily one... but I don't know if daily will continue to fail since it will be an insert and an update operation at the same time.... Any thoughts? Thank you


09/18/2023, 9:39 PM
are you and @Lori colleagues? 😆 close timing on these questions

Alec Ventura

09/18/2023, 9:45 PM
no haha
but if someone help him I will for sure keep my eye on that thread xD


09/18/2023, 11:09 PM
Alec and I asked the similar questions haha. I am feeling my spark conf does not work, but I do not know why.