Hello I have a Delta table that is defined by a path and is partitioned by time. Currently, when I use the Spark reader to count all the records in the table, it scans the file system and retrieves files in order to perform the count operation. However, I believe that the necessary information to determine the count of records already exists in the checkpoint.parquet file. I would like to know if there is a way to leverage the existing checkpoints and retrieve the count more efficiently, without the need to scan the file system or retrieve additional metadata (just use computed snapshot).

what version/connector are you using? this has been supported in the spark connector since 2.2 (https://github.com/delta-io/delta/releases/tag/v2.2.0)

Aggregate pushdown into Delta scan for SELECT COUNT(*). Aggregation queries such as SELECT COUNT(*) on Delta tables are satisfied using file-level row counts in Delta table metadata rather than counting rows in the underlying data files. This significantly reduces the query time as the query just needs to read the table metadata and could make full table count queries faster by 10-100x.

Yep, @Scott Sandre (Delta Lake) and I wrote a blog post discussing this with some benchmarks: https://delta.io/blog/2023-04-19-faster-aggregations-metadata/

I'm currently working with Delta Lake version 2.0.2 but previously, about 6 months ago, I tested the

dataframe.count()

operation and noticed that it provided fast results

So technically you need to scan the logs to read the count value, so there is no avoiding the file system