#dat
Title
n
Nick Karpov
05/15/2023, 6:20 PMthis should work
Copy code
val conf = spark.sparkContext.hadoopConfiguration
val fs = org.apache.hadoop.fs.FileSystem.get(conf)
fs.setWriteChecksum(false)m
Matthew Powers
05/15/2023, 6:52 PMYou know how to convert this to PySpark?
@Nick Karpov - your solution worked to suppress the Parquet CRC files, see the updated PR: https://github.com/delta-incubator/dat/pull/39
The CRC JSON files in the Delta Log are still being outputted.
257 Views