Hi folks, I am trying to run multiple concurrent stream writes to same location but with different partition, I am getting ProtocolChangedException: The protocol version of the Delta table has been changed by a concurrent update. I am running this in append mode with different checkpoint location & different partition but same location. any idea on how to handle this scenario?

hum that’s strange, compactions in the background maybe ? are you on Databricks ?

yeah,

autocompaction is on ?

I’m with default.

df.writeStream.format('delta').outputMode('append').option('checkpointLocation', <distinctfordifferentStreams>).PartitionBy(['snapshotTimestamp']).start('<same location>')

this is a known issue. If you have multiple concurrent streaming writers, it’s better to create the table before running them. Otherwise, all of these writers will try to create the table concurrently and you will see the error like

ProtocolChangedException

just confirming my understanding - we need to create table before initiating multiple concurrent stream. and that table will support concurrent writes.

Once you create the table before initiating multiple concurrent stream, the table should support append-only concurrent writes in any cases. See https://docs.delta.io/latest/concurrency-control.html for non append-only writes.

Will check it out. Thank you...