I have a Spark Structured Streaming application that is merging data into a target Delta Lake table using UpdateAll every microbatch. I had

mergeSchema=True

set on the DataFrameWriter and was surprised to see that the application wasn’t progressively merging in new columns. I looked up some documentation and noticed that for merges I need to set

spark.databricks.delta.schema.autoMerge.enabled=true

to enable schema evolution. I want to make sure I’m understanding the two configurations correctly: • mergeSchema=true will enable schema evolution for write operations •

spark.databricks.delta.schema.autoMerge.enabled=true

enables schema evolution for merge operations (InsertAll and UpdateAll)