https://delta.io logo
m

Michael Nacey

06/27/2023, 7:44 PM
Does anyone know of a situation that would cause this error to happen in batch write:
Copy code
user class threw exception: org.apache.spark.sql.delta.DeltaIllegalStateException: State of the checkpoint doesn't match that of the snapshot.
I found the source code that throws this error, but it doesn't seem possible for it to happen. Clearly it is, and the numberOfFiles check is in there for a reason. Could concurrent inserts cause this? Thanks.