Intialization stage is taking time in lakehouse pipeline

Anish_2 — Fri, 03 Jul 2026 22:01:20 GMT

Hello Team,

My intialization stage in lakehouse pipeline is taking 5-6 mins. When i checked event log table,below are stats

Message Reported flow time metrics for flowName: 'pipelines.flowTimeMetrics.missingFlowName'. is repeatedly coming for 4-5 mins. Can someone help what can be possible root cause for this message?

Re: Intialization stage is taking time in lakehouse pipeline

Ashwin_DSA — Sun, 05 Jul 2026 13:51:01 GMT

Hi @Anish_2,

The message... Reported flow time metrics for flowName: 'pipelines.flowTimeMetrics.missingFlowName'..is usually not enough on its own to identify the root cause. I would treat it as a metrics/observability signal rather than the primary failure, and I would focus on the surrounding pipeline events for the same update. The public docs describe the Lakeflow event log as the main source for execution progress, data quality, lineage, and streaming metrics, and they recommend using it to inspect pipeline behaviour in detail.

A good next step is to query the event log for the same update_id and look at the events immediately before and after these messages. Databricks documents the event_log('<pipeline-id>') table-valued function for this purpose in Pipeline event log and Monitor pipelines.

For example, you can review the latest flow progress events like this:

SELECT timestamp, level, event_type, message, details, origin
FROM event_log('<pipeline-id>')
WHERE origin.update_id = '<update-id>'
ORDER BY timestamp;

When reviewing that output, I would specifically check for these patterns:

WAITING_FOR_RESOURCES, autoscaling, or cluster resize activity, which can indicate the pipeline is spending time acquiring or resizing compute rather than actually processing data. The autoscaling docs note that scaling behaviour and worker limits can affect latency, and that classic pipelines expose autoscaling events in the event log.
SETTING_UP_TABLES, STARTING, RUNNING, stream_progress, or operation_progress events, which help confirm whether the pipeline is making progress but is simply slow to initialize. Databricks documents these monitoring surfaces in Monitor pipelines in the UI and Pipeline event log.
Any explicit error events in the same update, especially driver, source, schema, or permission-related failures. The monitoring UI and event log are the recommended places to inspect these details.

If the pipeline is using classic compute, a 5 - 6 minute initialisation can sometimes be explained by compute startup or autoscaling behaviour rather than by the missingFlowName message itself. Databricks recommends Enhanced autoscaling for Lakeflow pipelines and explains that compute configuration can influence latency and startup behaviour.

If this is a new pipeline or you are re-evaluating the compute setup, it is also worth reviewing Best practices for Lakeflow Spark Declarative Pipelines, which recommends using the event log for observability and discusses compute choices such as serverless and autoscaling.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

topic Re: Intialization stage is taking time in lakehouse pipeline in Data Engineering

Intialization stage is taking time in lakehouse pipeline

Re: Intialization stage is taking time in lakehouse pipeline