Yogasathyandrun
Contributor

This is a real limitation in the current Lakeflow / DLT job model.

Today, a pipeline is treated as the unit of refresh, not individual tables inside it. That means:

  • You can run or fully refresh a pipeline

  • But you cannot define different refresh policies per table in Jobs or DAB YAML

The REST API does provide additional flexibility via refresh_selection and full_refresh_selection, but this is only per execution, not something that can be declared and stored as part of a job definition.

Because of that, per-table scheduling or refresh policies inside a single pipeline are not supported today.

Common workarounds are:

  • Splitting pipelines by refresh frequency (high-frequency vs low-frequency tables)

  • Or using external orchestration that calls the REST API to selectively refresh subsets of tables per run

One partial alternative worth noting: materialized views and streaming tables in Databricks SQL can be scheduled independently, since each is backed by its own pipeline. That allows per-object refresh frequency, but it does not provide heterogeneous SLAs inside a single multi-table pipeline.

So overall, this is a real product gap rather than something configurable in Jobs or DAB today.

Data Engineer | Apache Spark | Delta Lake | Databricks