Adhoc Table Refresh in Lakeflow Spark Declarative Pipelines (SDP)

[thedatacrew]

Hi,

It is currently not possible to specify a list of tables to refresh and their refresh policies (full/normal) in a Lakeflow Job.

It can be done via the REST API, but it's messy.

For example, if you need some tables or views refreshed more regularly, the only option at the moment is to bake this into a notebook using the REST API and schedule/trigger the notebook.

It would be useful if you could specify a list of tables in a pipeline job in the ui or DAB yaml and their refresh policy.

Regards

Toby

[Mridu]

You’re right — this is a real gap today in Lakeflow / pipeline orchestration.

The current model is effectively “pipeline = unit of refresh”, not “table/view = unit of refresh with policy”, which makes mixed-frequency refresh use cases awkward.

What people are doing today (and why it feels clunky)

What you described is basically the de-facto workaround:

• Maintain a list of tables
• Call the REST API / pipeline refresh endpoints programmatically
• Encode full vs incremental (normal) logic per table
• Run that via a notebook/job

It works, but it introduces:

• Operational drift (logic hidden in notebooks instead of declarative config)
• Harder governance (no single source of truth like DAB YAML)
• Reduced observability (harder to map pipeline runs → table refresh behaviour)

So I agree — it’s “possible”, but not “productized”.

---

Why this limitation exists (likely design trade-off)

Lakeflow pipelines are fundamentally designed around:

Declarative graph execution (DAG) with dependency tracking, not imperative per-table scheduling.

So today:

• The system decides what needs recomputation based on lineage
• “Full vs normal refresh” is a pipeline-level override, not a per-node (table) policy

Adding per-table policies would effectively introduce:

• Mixed execution semantics within the same DAG
• Potential conflicts with dependency resolution (e.g., downstream tables depending on “less frequently refreshed” upstream nodes)

So I suspect that’s why the product hasn’t exposed this yet — it complicates the DAG contract.

---

What would actually make this usable (practical suggestion)

From a platform perspective, the feature that would help most is something like:

Option A: Declarative refresh policy in DAB YAML

resources:

  pipelines:

    my_pipeline:

      tables:

        - name: fact_sales

          refresh: normal

          schedule: "*/30 * * * *"

Option B: Tag-based policy (more scalable)

policies:

  - tag: high_frequency

    refresh: normal

    schedule: "*/15 * * * *"

  - tag: low_frequency

    refresh: full

    schedule: "0 3 * * *"

Attach tags at table level:

CREATE OR REFRESH TABLE dim_customer

TBLPROPERTIES ("refresh_policy" = "low_frequency")

This avoids hardcoding table lists and scales better across pipelines.

---

What I’ve seen work reasonably well in practice

Until something like this exists, a slightly cleaner pattern than raw REST-in-notebook is:

1. Externalize config (don’t hardcode in notebook)

Keep a config table or YAML:

table_name | refresh_type | frequency
-----------|--------------|-----------
fact_sales | normal       | 30 min
dim_customer| full        | daily

2. Build a lightweight orchestration layer

• Small driver notebook / job that:
  • Reads config
  • Groups tables by refresh policy
  • Calls pipeline refresh selectively (or triggers separate pipelines)

3. Split pipelines strategically

Instead of one monolithic pipeline:

• High-frequency pipeline (incremental tables)
• Low-frequency pipeline (dimension / heavy recompute)

This aligns better with how Lakeflow is designed and reduces friction.

---

Where the real gap still is

Even beyond UI/YAML convenience, I think the deeper gap is:

No first-class concept of heterogeneous refresh SLAs within a single pipeline DAG

That shows up in multiple ways:

• Can’t express SLA per dataset
• Can’t optimize cost vs freshness tradeoffs cleanly
• Forces either:
  • Over-refreshing everything, or
  • Splitting pipelines (operational overhead)

---

Bottom line

• You’re correct — not supported natively today
• REST workaround is common but non-ideal
• Splitting pipelines + external config is the least painful current approach
• There’s a clear product gap around per-table refresh policy + scheduling

If this lands natively (especially via DAB), it would significantly improve:

• Cost control
• SLA-driven design
• Pipeline maintainability

[thedatacrew]

Great feedback, Mridu. I think surfacing it in the job gives the most flexibility. You can do it in the pipeline refresh UI and in the REST API, so surfacing it in the Job YAML seems like a quick win.

i.e.

resources:
  jobs:
    dataverse_data_job:
      name: data_load_${bundle.target}
      email_notifications:
        on_failure:
          - support@xxx.com
      max_concurrent_runs: 2
      tasks:
        - task_key: load_bronze
          pipeline_task:
            pipeline_id: ${resources.pipelines.bronze_pipeline.id}
            full_refresh: true       

        - task_key: load_silver
          depends_on:
            - task_key: load_bronze
          pipeline_task:
            pipeline_id: ${resources.pipelines.silver_pipeline.id}
            full_refresh: true
          tables:
            - name: schema.table_1
              refresh_policy: incremental or full
            - name: schema.table_2
              refresh_policy: incremental or full

      queue:
        enabled: true
      permissions:
        - service_principal_name: ${var.service_principal_pipeline_and_jobs_owner}
          level: IS_OWNER

[Yogasathyandrun]

This is a real limitation in the current Lakeflow / DLT job model.

Today, a pipeline is treated as the unit of refresh, not individual tables inside it. That means:

• You can run or fully refresh a pipeline

• But you cannot define different refresh policies per table in Jobs or DAB YAML

The REST API does provide additional flexibility via refresh_selection and full_refresh_selection, but this is only per execution, not something that can be declared and stored as part of a job definition.

Because of that, per-table scheduling or refresh policies inside a single pipeline are not supported today.

Common workarounds are:

• Splitting pipelines by refresh frequency (high-frequency vs low-frequency tables)

• Or using external orchestration that calls the REST API to selectively refresh subsets of tables per run

One partial alternative worth noting: materialized views and streaming tables in Databricks SQL can be scheduled independently, since each is backed by its own pipeline. That allows per-object refresh frequency, but it does not provide heterogeneous SLAs inside a single multi-table pipeline.

So overall, this is a real product gap rather than something configurable in Jobs or DAB today.