Alber Tadrous
06/05/2023, 3:34 PMTheo LEBRUN
06/05/2023, 4:15 PMoption("mergeSchema", "true")
.
To store the schema definition, especially if it’s big, I recommend storing it as JSON and then loading it using StructType.fromJson
(https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.types.StructType.html#pyspark.sql.types.StructType.fromJson).Alber Tadrous
06/05/2023, 6:20 PM