https://delta.io logo
a

Alber Tadrous

06/05/2023, 3:34 PM
Hi team, What is the best way to manage delta table schemas specially if these schemas might change down the road?
Also, what is better to create a sql schema for our delta tables or should we use pyspark "StructType, StructFields"
t

Theo LEBRUN

06/05/2023, 4:15 PM
Delta supports schema evolution so you can just merge the new schema every time you have changes using
option("mergeSchema", "true")
. To store the schema definition, especially if it’s big, I recommend storing it as JSON and then loading it using
StructType.fromJson
(https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.types.StructType.html#pyspark.sql.types.StructType.fromJson).
a

Alber Tadrous

06/05/2023, 6:20 PM
Thank you
2 Views