Hello, how can we get a record count for each partitioned value of delta table without reading the delta table (select count). I was looking around deltaLog.

when you say without reading the delta table, do you mean you don't want to load the table at all, or you don't want to pull the full contents to memory? If the latter, running select count and grouping by the partition variable will only ready the metadata and should be very fast

there’s been a lot of discussion about adding a nice high level API for this, feel free to submit a github issue. it would be nice to be able to load the metadata with a direct API

yeah I was thinking about trying to add a python wrapper that just gives you add actions from the delta log as a dataframe. you can hack around it to get it from the jvm to get super quick stats from the delta log. can also get it from scala land theoretically too, but not an official public API

yeah all this info is in the

snapshot

object, but its an internal class, not exposed as an easy-to-use public API

you can try approximate functions if that works .. https://spark.apache.org/docs/3.4.1/api/sql/#approx_count_distinct https://spark.apache.org/docs/3.4.1/api/sql/#approx_percentile