Hello Everyone,
Please help me with the below scenario?
Scenario-
RawDf written to delta table with schema - Id, Name, Age, Salary
Incremental data comes in with an additional column - Id, Name, Age, Salary, Designation
I'm using merge as below:
targetTable.alias("tgt").merge(df.alias("src"), condition = "tgt.id = src.mergeKey and tgt.iscurrent = true").whenMatchedUodate(set = {"enddate ": currenttimestamp(), "isCurrent ": "false"}).whenNotMatchedInsertAll().execute()
Spark.conf.set("spark.databricks.delta.schema.autoMerge.enabled", True) --> this is also enabled
But new column I..e Designation not inserted. Am I doing anything wrong here...please help.
Thanks!