Hi All, Is the delta merge function async? Means it will execute the next line of code without waiting for the merge operation to complete.

Typically, things are done in order. If you’re using spark and submit an application, it doesn’t run it all at once but from the top down.

Definitely not async. why do you ask?

@Dominique Brezinski I observed this while running spark on AWS lambda and it has spark along with delta library. The scenario where we were running a merge on the dataframe into S3 delta format table and in the next step we are putting records in the dynamodb table using the same dataframe. I observed that data was not in the delta table but present in Dynamodb table.

That is likely because your merge conditions did not addend or update data

But it should add the data into the S3 delta table....and then write to ddb... However I was not able to deep dive into it

It won’t add the data to the s3 delta table if the merge logic does not trigger an insert, update, or delete. I am saying the merge completed before the ddb write, but the merge did not do what you were expecting I suspect.

Okay .... Understood

That is my best guess. Spark code runs sequentially.

maybe you simply forgot the

.execute()

at the end of your merge function? (happened to me once and took me ages to find it)

I used .execute()