Laurens Janssen

06/22/2023, 11:22 AM
I was wondering if this library uses metadata statistics embedded in the Delta log for file filtering? I did a quick read of the code and noticed that file filtering functionality is usually transferred to the PyArrow library. But doesn't this imply that each parquet file is opened to read the parquet statistics?