Hi @Matthew Powers, thanks for the great blog post on partitions: https://delta.io/blog/2023-01-18-add-remove-partition-delta-lake/ There is this code snipped in the post:

df.repartition(F.col("country")).write.partitionBy("country").format(
    "delta"
).saveAsTable("country_people")

I was wondering: is there a benefit of performing an explicit

repartition

on the partition column before writing into a partitioned table? Delta does the repartition implicitly anyhow, doesn't it?

Let me do some digging and report back.

in that code block the benefit of the repartition on the dataframe prior to writing is to ensure that all rows of any given country are colocated in the same partition

Yea, think that

repartition(F.col("country"))

was mainly just to output a single file to make the example cleaner. But I should probably note that in the blog.

thanks guys!

due to the shuffling of

repartition()

I guess it can be slightly slower but ensures that you write the minimum amount of files. Would be interesting if

optimizeWrite

would consolidate the files in a similar way as an explicit

repartition

🤔

that's exactly what it does 😄 will be exciting to see the PR merged

So is doing optimizeWrite just doing a repartition in the background?

simply put yes, but a little fancier https://github.com/delta-io/delta/pull/1198

Nice!