Hey question about Data Skipping - does collecting statistics happen when writing to delta? or on the optimize job? attaching quotes from databricks delta documentation -

By default Delta Lake on Databricks collects statistics on the first 32 columns defined in your table schema

stats are collected when writing data

@JosephK (exDatabricks) so if i got it right, it also means that when reducing the value of

delta.dataSkippingNumIndexedCols

for an hour and then increase it back again the columns that were out of this range on that hour will never get their statistics back, right?

First, don't do that. They wont' have stats but if you optimize, that will write new files and you'll get stats for writing

not sure i understand.. you mean when increasing the value of this property back, it will write the (missing) stats on the optimize job?

If you change it back, the optimize will create newer stats

ok so it can be done also on the optimize not only when writing the data

optimize writes files