Does anyone know if it is possible to track stats for datafusion delta-rs queries? Basically I want to see how many files where considered, pruned and then how many bytes fetched from the object store.
06/01/2023, 5:47 AM
Not via "first class" support so far, but this is definitely on my todo list. Once an execution plan is executed, you can inspect the metrics generated by it. Many metrics that the datafusion operators supply could already be gathered, but I wanted to look into generating custom metrics more related to delta as well.