Yogasathyandrun
New Contributor

A few details would help narrow this down.

When the scheduled run executes:

  1. Does the pipeline update show Succeeded or Failed?

  2. In the pipeline Event Log, do you see rows being processed/written?

  3. Is your manual run a normal update or a Full Refresh?

  4. Is the schedule configured directly on the DLT pipeline or triggered through a Workflow/Job?

Common causes are:

  • No new data available at the scheduled time (the update succeeds but processes 0 rows).

  • Incremental processing vs Full Refresh (manual full refresh reprocesses historical data, scheduled updates only process new data).

  • Different configuration/parameters between scheduled and manual runs.

  • Permission differences if the pipeline is being triggered through another service or workflow.

The first thing I'd check is the Event Log for a scheduled run and compare it to a successful manual run. If both succeed but the scheduled run shows 0 records processed, the issue is usually related to source data availability or incremental processing rather than the pipeline logic itself.

Data Engineer | Apache Spark | Delta Lake | Databricks