Hi Team,
I store my delta tables schema definition in a central repo in json format, to enforce and validate schema of the incoming data, but when I tried to update a column data type from Timestamp to string using this commands:
struct = StructType().fromJson(new_schema_string)
df2 = spark.createDataFrame(data=[],schema=struct)
df2.write.mode("append").format("delta").option("overwriteSchema", "true").saveAsTable("table_name")
it throws this error:
Failed to merge fields 'createdt' and 'createdt'. Failed to merge incompatible data types TimestampType and StringType
what should I do?