Hello Team I have 2 spark jobs one keeps adding new set of r

Question

Hello Team I have 2 spark jobs one keeps adding new set of rows to a delta table and other performs updates on this same delta table Inserts are more frequent than the updates Inserts are done by spark write commands and updates using spark sql We are seeing that that some of the updates are getting lost once in hundred runs and we don t get any exceptions Has anyone observed similar issue and mitigated it Delta v2 0 1

Chetan Joshi · Answer

More details Delta table schema gt inputId processedFlag fileName metadata Inserts are done by writing data partitioned by incremental inputId by job1 Updates are done by job2 to set processedFlag=true using following where condition gt `fileName in file1 file2 and inputId lt N` basically we are marking files which got processed as true and taking max inputId N corresponding to these files to avoid partition file lookup for new entries

Madhumita Bharde · Answer

delta on S3

Chetan Joshi · Answer

It is over Oracle Cloud s Object Store

Madhumita Bharde · Answer

Madhumita Bharde · Answer

it essentially is supposed to be issue with consistency guarantees provided by underlying object storage Don t know much about Oracle Cloud s Object Store

Chetan Joshi · Answer

Thanks let me take a look

Nick Karpov · Answer

it s likely you ll need to implement a LogStore very similar to <https delta io blog 2022 05 18 multi cluster writes to delta lake storage in s3 |this one> one but with an Oracle service I m assuming dynamodb is not an option perhaps oracle s nosql