bianca_unifeye
Databricks MVP

Please read https://docs.databricks.com/aws/en/ldp/monitor-event-logs

Lakeflow ingestion pipelines do generate event logs, but they behave differently than classic Spark Declarative Pipelines. 

 

✔ Lakeflow Connect pipelines include event logs as part of observability/monitoring
✔ The event log contains things like audit info, progress, data quality, etc,  the same kinds of logs you expect for DLT.

By default:

  • The event log is written as a hidden Delta table in the pipeline’s default catalog/schema.

  • Only the pipeline owner can query it.

  • This is by design, not a missing feature. 

    You won’t find a system.lakeflow table source documented, the doc approach is to use event log directly or publish it.

     

    Even when it’s hidden, you can still query it as the owner using the pipeline ID via:

     

    SELECT * FROM event_log(<pipelineId>);

    Databricks documents this exact approach, including the default hidden naming convention and owner-only access by default.

View solution in original post