https://delta.io logo
r

Ryan Aston

09/09/2023, 9:21 PM
Hi all, I have a question about partition values. One of my partition columns is a file path which has forward-slashes. It seems to work fine, but I want to make sure there aren’t any gotchas with this. As an example, the partition values for a particular Parquet file are
{"received_date": null, "context_page_path": "/bar/"}
and the resulting path in S3 after the write is
<s3://bucket/table/received_date=__HIVE_DEFAULT_PARTITION__/context_page_path=/bar/part-00000-da65176e-fb16-418b-be85-b3a28d3edc11-c000.snappy.parquet>
. It seems like if
context_page_path
was ever
/bar
it would collide with this partition, even though it’s technically a different value.