Hi everyone, we are getting the "error" as we ran streaming API: • "org.apache.spark.sql.execution.streaming.state.StateSchemaNotCompatible: Provided schema doesn't match to the schema for existing state! Please note that Spark allow difference of field name: check count of fields and data type of each field." What are the consequences of setting the

set spark.sql.streaming.stateStore.stateSchemaCheck to false

? Main goal would be to merge schemas while streaming (adding new columns, ...). When running this code in Notebook (12.2.x-scala2.12) it works as opposed to Job (10.4.x-scala2.12). Was it some change done? Thanks and regards, GapyNi

Not a delta lake question. You should contact your support. Streaming doesn't support schema change

Hi Joseph, thanks for the answer, however do i understand it correctly, that it says in the following article that schema evolution is supported with writeStream https://www.databricks.com/blog/2019/09/24/diving-into-delta-lake-schema-enforcement-evolution.html

writing and reading aren't the same thing

Hi Joseph, could you explain it a bit more? Thanks

Streaming requires a fixed schema for reading and writing supports schema evolution.