Lakeflow Ingestion Pipeline – Unable to Generate Event Log Table

[agent007]

For the DLT pipeline, we are successfully generating an event log table using the event_log configuration in the pipeline YAML.

However, for the Salesforce pipeline, which is an ingestion pipeline (via Lakeflow Connector), we are unable to create or access a similar event log table.

---

What We Tried

We attempted enabling event logs in the Salesforce pipeline YAML configuration.

However:

• The event log table is not being created
• We can see pipeline logs in the Databricks UI (screenshot attached)
• But we cannot find or query any corresponding event log table
  
  

  ---

  Questions
  1. Is event log table creation supported for ingestion pipelines (Lakeflow Connector)?
  2. If yes, what is the correct configuration to enable and persist event logs as a Unity Catalog table?
  3. Is there anything we could find in system.lakeflow (just an assumption as we don't have access to it)
  4. If direct event log tables are not supported for ingestion pipelines, what would be the recommended alternative approach to expose pipeline execution logs for reporting in Power BI?
     
     Attached Image - 
  

[bianca_unifeye · Accepted solution]

Please read https://docs.databricks.com/aws/en/ldp/monitor-event-logs

Lakeflow ingestion pipelines do generate event logs, but they behave differently than classic Spark Declarative Pipelines. 

 

✔ Lakeflow Connect pipelines include event logs as part of observability/monitoring
✔ The event log contains things like audit info, progress, data quality, etc,  the same kinds of logs you expect for DLT.

By default:

• The event log is written as a hidden Delta table in the pipeline’s default catalog/schema.

• Only the pipeline owner can query it.

• This is by design, not a missing feature. 

  You won’t find a system.lakeflow table source documented, the doc approach is to use event log directly or publish it.

  Even when it’s hidden, you can still query it as the owner using the pipeline ID via:

  SELECT * FROM event_log(<pipelineId>);

  Databricks documents this exact approach, including the default hidden naming convention and owner-only access by default.