- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-23-2026 04:32 AM
This is a real limitation in the current Lakeflow / DLT job model.
Today, a pipeline is treated as the unit of refresh, not individual tables inside it. That means:
You can run or fully refresh a pipeline
But you cannot define different refresh policies per table in Jobs or DAB YAML
The REST API does provide additional flexibility via refresh_selection and full_refresh_selection, but this is only per execution, not something that can be declared and stored as part of a job definition.
Because of that, per-table scheduling or refresh policies inside a single pipeline are not supported today.
Common workarounds are:
Splitting pipelines by refresh frequency (high-frequency vs low-frequency tables)
Or using external orchestration that calls the REST API to selectively refresh subsets of tables per run
One partial alternative worth noting: materialized views and streaming tables in Databricks SQL can be scheduled independently, since each is backed by its own pipeline. That allows per-object refresh frequency, but it does not provide heterogeneous SLAs inside a single multi-table pipeline.
So overall, this is a real product gap rather than something configurable in Jobs or DAB today.