I am getting incredibly slow performance on a table with 30k partitions, is that normal?
j
Jeremy Jordan
08/18/2023, 1:08 PM
we were observing poor performance as a result of overpartitioning our table. how large are the files in each of those partitions? performance suffers when you have too many small files
💯 1
i
Ion
08/18/2023, 1:12 PM
Yeah I figured the same, the source table that I created it from had also the same amount of partitions, but that one was 60tb. This one is way less, so i've removed one partition by column that blew up all the partitions