Hello everyone. Recently, we have received a data stream from Kafka, which contains several fields in the form of text with very large lengths. I was wondering if anyone has any solutions to optimize storage for this data.

Parquet, which Delta uses, supports compression. Snappy is the default, but you can change this when you save the files or at the cluster level. https://spark.apache.org/docs/latest/sql-data-sources-parquet.html#data-source-option

thank you