raphaelblg
Databricks Employee
Databricks Employee

@dmytro yes, it's possible to monitor the consumer lag through the streaming query metrics. Every cluster that runs a spark structured streaming query will log the metrics for each streaming batch in the driver logs and Spark UI. More details at Monitoring Structured Streaming queries on Databricks.

Best regards,

Raphael Balogo
Sr. Technical Solutions Engineer
Databricks