szymon_dybczak
Esteemed Contributor III

Hi @ajgold ,

I think I have some idea for workaround. Lakeflow Declarative Pipelines (former DLT) generate event log that contains all information related to pipeline like:

- audit logs

- data quality checks (this is what you're looking for)

- progress

- data lineage

Lakeflow Declarative Pipelines writes the event log to a hidden Delta table in the default catalog and schema configured for the pipeline. While hidden, the table can still be queried by all sufficiently privileged users. By default, only the owner of the pipeline can query the event log table. 

You can read how to query event_log here:

Monitor Lakeflow Declarative Pipelines - Azure Databricks | Microsoft Learn

And you can read how to query data quality from the event log here:

Monitor Lakeflow Declarative Pipelines - Azure Databricks | Microsoft Learn

 

Now, you can configure custom monitoring for lakeflow declartive pipelines (former DLT) with event hooks. This allow you to send notification to your Slack, Teams etc. Here's a relevant documentation entry to read.

So to sum it up, assuming that event_log contains the information you're looking for you can try to define event hook for that type of event and send message to your client of choice.

Define custom monitoring of Lakeflow Declarative Pipelines with event hooks - Azure Databricks | Mic...