Divyansh Jain

06/02/2023, 1:21 PM
Hello Everyone, Please help me with the below scenario? Scenario- RawDf written to delta table with schema - Id, Name, Age, Salary Incremental data comes in with an additional column - Id, Name, Age, Salary, Designation I'm using merge as below: targetTable.alias("tgt").merge(df.alias("src"), condition = " = src.mergeKey and tgt.iscurrent = true").whenMatchedUodate(set = {"enddate ": currenttimestamp(), "isCurrent ": "false"}).whenNotMatchedInsertAll().execute() Spark.conf.set("", True) --> this is also enabled But new column I..e Designation not inserted. Am I doing anything wrong here...please help. Thanks!

Rahul Sharma

06/02/2023, 5:37 PM
try to set below property
Copy code
.option("mergeSchema", "true")				.option("overwriteSchema", "true")