Hello Everyone, Please help me with the below scenario? Scenario- RawDf written to delta table with schema - Id, Name, Age, Salary Incremental data comes in with an additional column - Id, Name, Age, Salary, Designation I'm using merge as below: targetTable.alias("tgt").merge(df.alias("src"), condition = "tgt.id = src.mergeKey and tgt.iscurrent = true").whenMatchedUodate(set = {"enddate ": currenttimestamp(), "isCurrent ": "false"}).whenNotMatchedInsertAll().execute() Spark.conf.set("spark.databricks.delta.schema.autoMerge.enabled", True) --> this is also enabled But new column I..e Designation not inserted. Am I doing anything wrong here...please help. Thanks!

does your source actually contain any new data that would be inserted/updated? and if yes, does the new column have an actual value?

Attaching the snippets for your reference. Existing Delta Table looks like below: Image --> SS1 Now I have an incremental data with an additional column i.e. owner: Dataframe Name --> scdDF Image --> SS2 Below is the code snippet to merge Incremental Dataframe to targetTable, but the new column is not getting added: Image --> SS3 spark.conf.set(“spark.databricks.delta.schema.autoMerge.enabled”,True) --> this is also enabled. But still no luck. Below is the final result which i’m currently getting: Image --> SS4 Data looks correct, but the only issue is New Column i.e. Owner column is still not merged in targetTable.

not 100% sure if schema evolutio also works if it is "only" an update but not an insert - but it should I think whats a bit weird is is that your MergeKey column contains NULL for one of the ID/Landmark combination you want to insert which might also be part of the issue