I was performing benchmarking for open table formats against some parameters. I created two tables, one is in iceberg format(parquet) and another in delta lake using spark-3.3(emr-6.9). In the initial data load (insert) into the table operation, iceberg output data volume was 102GBs against input data volume of 161GBs. While on the delta lake the output data volume was 228GBs against the same input data volume of 161GBs. Do anyone know the reason?

What is the compression format for both?

Input data is in snappy, iceberg dataset is in gzip and delta lake dataset is in snappy

Well, there is your answer

But in delta lake, does it suppose to explode from 161GBs to 228GBs since both input and output are in same compression format?

Depends on the input file format

Exactly, the vacuum did nothing. Okay I’ll give it a try and compare it with iceberg using the same compression format.

Vacuum deletes the old files in the history. The history has only a single version, write, so there is no history.

Although both the tables had significant difference in data volume but the query against these tables were taking almost same time to execute.

That’s good. Delta should outperform Iceberg. There are many third party benchmarks that show this. https://databeans-blogs.medium.com/delta-vs-iceberg-vs-hudi-reassessing-performance-cb8157005eb0. Don’t believe anything by us, tabular, or onehouse

I configured the compression to snappy for iceberg table. The output data volume is 184GBs against input data volume of 161GBs. It is still better than Delta lake (228GBs).

This topic seems familiar...

Initial data load query execution time is also different. In Iceberg it took ~28mins while in delta lake it took ~55 mins.

What was the initial file format?

Parquet, snappy

So the sizes shouldn’t change so much if you’re reading parquet snappy and writing parquet snappy. All you’re adding is some logs, which on the big side tens of MBs

Yeah, so as I mentioned earlier I was using default iceberg compression for parquet (i.e. gzip). But later I changed it to snappy and still it is better than delta lake. Although overall the size has been increased for both iceberg and delta table.

The operation is just spark.read.parquet(path).write.format(delta/iceberg).save(path2)

For iceberg I used DF writer V2 apis. So it was df.format(“iceberg”).writeTo(table). I don’t think this makes any difference.

The V2 writer might affect things. Not sure what writer delta uses on emr