Hi folks, I was wondering if there was a way to collect column statistics from delta tables using spark? I am specifically interested in the distinct values. My current presto cluster is dependant on those stats for optimizations

Delta is just files on disk, so you can read those files and compute statistics. There is a count_distinct() function in spark sql. The delta_log doesn’t compute distinct stats because those don’t help with file skipping.

@JosephK (exDatabricks) is there a specific connector for spark that will allow me to read only the active files while computing stats in the Analyze table command? I need these stats to be populated in my metastore for the query optimizer.

You can use spark to read a table and not need any particular connector:

spark.read.format("delta").load(path)