Using Python `deltalake` with Pyarrow and S3 and trying to m

Question

Using Python `deltalake` with Pyarrow and S3 and trying to minimize query latency I noticed that Pyarrow makes HEAD requests for every file it reads I believe this is to determine the file size so the parquet footer can be read However I think the delta log already contains the file size so in principle this could be communicated to Arrow to avoid the HEAD requests see e g <https github com apache arrow pull 7547> Has anyone looked into something like this

Eero Lihavainen · Answer

Or maybe a more relevant question is is this kind of optimization done in the `delta rs` s3 filesystem implementation

Will Jones · Answer

We don t do that optimization right now no

Will Jones · Answer

Haven t looked into it since I plan on switching away from using PyArrow later anyways

Eero Lihavainen · Answer

Is that the responsibility of Delta Kernel or something else

Will Jones · Answer

Yeah might be something we could incorporate into Delta Kernels It s a good idea to pass that info down to the Parquet reader

Eero Lihavainen · Answer

I played with this idea a bit adding a `size` argument to the `get input file` file system method <https github com delta io delta rs pull 1630> With this addition the DeltaFileSystemHandler can be extended on Python side to pass the file sizes from the DeltaTable object Do you think this would be a reasonable way to do it

Will Jones · Answer

Well the base class doesn t have that parameter <https arrow apache org docs python generated pyarrow fs FileSystem html pyarrow fs FileSystem open input file|docs> so how would PyArrow know to pass the size down

Will Jones · Answer

It s PyArrow not delta rs that s calling `open input file `

Eero Lihavainen · Answer

Yeah sorry I wasn t clear I didn t add it to the draft but this parameter allows wrapping the method in the python implementation of DeltaFileSystemHandler So you can get file sizes using get add actions on the DeltaTable and pass them based on the path that pyarrow calls the method with Not sure if this makes sense though or if it would be better to do on the Rust side And are there cases when this HEAD request should not be skipped

Will Jones · Answer

Oh hmm `get add actions` isn t the most efficient way to get that but might be the only way that the add actions are exposed at the moment thinking face

Will Jones · Answer

Yeah that makes sense I guess DeltaStorageHandler could have a field `known sizes Dict str int ` that you can populate using `get add actions` here <https github com delta io delta rs blob ef17f9e87bf7028b6dd00df27549149c60282e02 python deltalake table py L525>

Will Jones · Answer

And then alter the filesystem to try to use `known sizes` instead of making a HEAD request

Eero Lihavainen · Answer

Opened PR with the complete implementation <https github com delta io delta rs pull 1630>