topic Adhoc Table Refresh in Lakeflow Spark Declarative Pipelines (SDP) in Data Engineering

Adhoc Table Refresh in Lakeflow Spark Declarative Pipelines (SDP)

thedatacrew — Wed, 17 Jun 2026 17:57:27 GMT

Hi,

It is currently not possible to specify a list of tables to refresh and their refresh policies (full/normal) in a Lakeflow Job.

It can be done via the REST API, but it's messy.

For example, if you need some tables or views refreshed more regularly, the only option at the moment is to bake this into a notebook using the REST API and schedule/trigger the notebook.

It would be useful if you could specify a list of tables in a pipeline job in the ui or DAB yaml and their refresh policy.

Regards

Toby

Re: Adhoc Table Refresh in Lakeflow Spark Declarative Pipelines (SDP)

Mridu — Tue, 23 Jun 2026 04:17:11 GMT

You’re right — this is a real gap today in Lakeflow / pipeline orchestration.

The current model is effectively “pipeline = unit of refresh”, not “table/view = unit of refresh with policy”, which makes mixed-frequency refresh use cases awkward.

What people are doing today (and why it feels clunky)

What you described is basically the de-facto workaround:

Maintain a list of tables
Call the REST API / pipeline refresh endpoints programmatically
Encode full vs incremental (normal) logic per table
Run that via a notebook/job

It works, but it introduces:

Operational drift (logic hidden in notebooks instead of declarative config)
Harder governance (no single source of truth like DAB YAML)
Reduced observability (harder to map pipeline runs → table refresh behaviour)

So I agree — it’s “possible”, but not “productized”.

Why this limitation exists (likely design trade-off)

Lakeflow pipelines are fundamentally designed around:

Declarative graph execution (DAG) with dependency tracking, not imperative per-table scheduling.

So today:

The system decides what needs recomputation based on lineage
“Full vs normal refresh” is a pipeline-level override, not a per-node (table) policy

Adding per-table policies would effectively introduce:

Mixed execution semantics within the same DAG
Potential conflicts with dependency resolution (e.g., downstream tables depending on “less frequently refreshed” upstream nodes)

So I suspect that’s why the product hasn’t exposed this yet — it complicates the DAG contract.

What would actually make this usable (practical suggestion)

From a platform perspective, the feature that would help most is something like:

Option A: Declarative refresh policy in DAB YAML

resources:

pipelines:

my_pipeline:

tables:

- name: fact_sales

refresh: normal

schedule: "*/30 * * * *"

Option B: Tag-based policy (more scalable)

policies:

- tag: high_frequency

refresh: normal

schedule: "*/15 * * * *"

- tag: low_frequency

refresh: full

schedule: "0 3 * * *"

Attach tags at table level:

CREATE OR REFRESH TABLE dim_customer

TBLPROPERTIES ("refresh_policy" = "low_frequency")

This avoids hardcoding table lists and scales better across pipelines.

What I’ve seen work reasonably well in practice

Until something like this exists, a slightly cleaner pattern than raw REST-in-notebook is:

1. Externalize config (don’t hardcode in notebook)

Keep a config table or YAML:

table_name | refresh_type | frequency
-----------|--------------|-----------
fact_sales | normal       | 30 min
dim_customer| full        | daily

2. Build a lightweight orchestration layer

Small driver notebook / job that:
- Reads config
- Groups tables by refresh policy
- Calls pipeline refresh selectively (or triggers separate pipelines)

3. Split pipelines strategically

Instead of one monolithic pipeline:

High-frequency pipeline (incremental tables)
Low-frequency pipeline (dimension / heavy recompute)

This aligns better with how Lakeflow is designed and reduces friction.

Where the real gap still is

Even beyond UI/YAML convenience, I think the deeper gap is:

No first-class concept of heterogeneous refresh SLAs within a single pipeline DAG

That shows up in multiple ways:

Can’t express SLA per dataset
Can’t optimize cost vs freshness tradeoffs cleanly
Forces either:
- Over-refreshing everything, or
- Splitting pipelines (operational overhead)

Bottom line

You’re correct — not supported natively today
REST workaround is common but non-ideal
Splitting pipelines + external config is the least painful current approach
There’s a clear product gap around per-table refresh policy + scheduling

If this lands natively (especially via DAB), it would significantly improve:

Cost control
SLA-driven design
Pipeline maintainability

Re: Adhoc Table Refresh in Lakeflow Spark Declarative Pipelines (SDP)

thedatacrew — Tue, 23 Jun 2026 11:07:44 GMT

Great feedback, Mridu. I think surfacing it in the job gives the most flexibility. You can do it in the pipeline refresh UI and in the REST API, so surfacing it in the Job YAML seems like a quick win.

i.e.

resources: jobs: dataverse_data_job: name: data_load_${bundle.target} email_notifications: on_failure: - support@xxx.com max_concurrent_runs: 2 tasks: - task_key: load_bronze pipeline_task: pipeline_id: ${resources.pipelines.bronze_pipeline.id} full_refresh: true - task_key: load_silver depends_on: - task_key: load_bronze pipeline_task: pipeline_id: ${resources.pipelines.silver_pipeline.id} full_refresh: true tables: - name: schema.table_1 refresh_policy: incremental or full - name: schema.table_2 refresh_policy: incremental or full queue: enabled: true permissions: - service_principal_name: ${var.service_principal_pipeline_and_jobs_owner} level: IS_OWNER

Re: Adhoc Table Refresh in Lakeflow Spark Declarative Pipelines (SDP)

Yogasathyandrun — Tue, 23 Jun 2026 11:32:42 GMT

This is a real limitation in the current Lakeflow / DLT job model.

Today, a pipeline is treated as the unit of refresh, not individual tables inside it. That means:

You can run or fully refresh a pipeline
But you cannot define different refresh policies per table in Jobs or DAB YAML

The REST API does provide additional flexibility via refresh_selection and full_refresh_selection, but this is only per execution, not something that can be declared and stored as part of a job definition.

Because of that, per-table scheduling or refresh policies inside a single pipeline are not supported today.

Common workarounds are:

Splitting pipelines by refresh frequency (high-frequency vs low-frequency tables)
Or using external orchestration that calls the REST API to selectively refresh subsets of tables per run

One partial alternative worth noting: materialized views and streaming tables in Databricks SQL can be scheduled independently, since each is backed by its own pipeline. That allows per-object refresh frequency, but it does not provide heterogeneous SLAs inside a single multi-table pipeline.

So overall, this is a real product gap rather than something configurable in Jobs or DAB today.