https://delta.io logo
l

Lori

09/18/2023, 7:01 PM
Hello team, I have a question for Delta Lake Multi-writer. I followed the Delta 2.2 Multi-writer setup steps to enable the Delta Lake Multi-writer for S3. However, after following these steps, I still see the
ConcurrentAppendException
when I write to the same table (use Delta
merge
) from 2 EMR clusters (EMR version 6.10.0, Spark 3.3.1, Delta 2.2.0), and there was no entries in the DynamoDB after writing. It looks like I did not enable Delta Lake Multi-writer successfully. My questions are: 1. How can I debug this? I checked online, but I did not find the useful solutions/debug steps 2. I use the
SparkSession.builder().appName().config(…)
to config the spark session. Is this correct? Besides the spark configuration listed in the Setup steps, is there any required Spark configurations I need to setup? Thank you!