So initially I thought my read execution time was slow due t

Question

So initially I thought my read execution time was slow due to my partitioning but it s apparently due to delta rs because passing all the file uris to pyarrow parquetdataset directly is 10x faster

Will Jones · Answer

hmm I need to look into this It shouldn t be 10x slower than PyArrow After all it uses PyArrow s parquet readers under the hood

Ion · Answer

This is the related issue <https github com delta io delta rs issues 1569|https github com delta io delta rs issues 1569>

Will Jones · Answer

Are you also using azure

Will Jones · Answer

Or is this reproducible on S3 or GCS

Ion · Answer

Our company only uses azure so I don t know about the others

Ion · Answer

Also in localfilesystem which I ve checked now with smaller partitions Delta RS is consistently slower and the gap grows a lot by the amount of partitions