Hi All, I need your help on below scenario. Kindly help me to clear on this case. In Databricks notebook, I am consuming data from Kafka to delta lake using Pyspark. If suppose I want to know the consumer lag , consumer group details means how I can find these details using pyspark or atleast do we have any other way to do it? My Databricks run time version: Databricks 7.5 (includes Apache Spark 3.0.1, Scala 2.12) Thanks in advance!

You have to setup separate scripts to monitor your kafka cluster. This is what I got from google search: • https://www.entechlog.com/blog/kafka/monitoring-kafka-consumer-lag/

Thanks I will look into it