You’re right — this is a real gap today in Lakeflow / pipeline orchestration.

The current model is effectively “pipeline = unit of refresh”, not “table/view = unit of refresh with policy”, which makes mixed-frequency refresh use cases awkward.

What people are doing today (and why it feels clunky)

What you described is basically the de-facto workaround:

• Maintain a list of tables
• Call the REST API / pipeline refresh endpoints programmatically
• Encode full vs incremental (normal) logic per table
• Run that via a notebook/job

It works, but it introduces:

• Operational drift (logic hidden in notebooks instead of declarative config)
• Harder governance (no single source of truth like DAB YAML)
• Reduced observability (harder to map pipeline runs → table refresh behaviour)

So I agree — it’s “possible”, but not “productized”.

Why this limitation exists (likely design trade-off)

Lakeflow pipelines are fundamentally designed around:

Declarative graph execution (DAG) with dependency tracking, not imperative per-table scheduling.

So today:

• The system decides what needs recomputation based on lineage
• “Full vs normal refresh” is a pipeline-level override, not a per-node (table) policy

Adding per-table policies would effectively introduce:

• Mixed execution semantics within the same DAG
• Potential conflicts with dependency resolution (e.g., downstream tables depending on “less frequently refreshed” upstream nodes)

So I suspect that’s why the product hasn’t exposed this yet — it complicates the DAG contract.

What would actually make this usable (practical suggestion)

From a platform perspective, the feature that would help most is something like:

Option A: Declarative refresh policy in DAB YAML

Option B: Tag-based policy (more scalable)

What I’ve seen work reasonably well in practice

Until something like this exists, a slightly cleaner pattern than raw REST-in-notebook is:

1. Externalize config (don’t hardcode in notebook)

Keep a config table or YAML:

table_name | refresh_type | frequency
-----------|--------------|-----------
fact_sales | normal       | 30 min
dim_customer| full        | daily

2. Build a lightweight orchestration layer

• Small driver notebook / job that:
  • Reads config
  • Groups tables by refresh policy
  • Calls pipeline refresh selectively (or triggers separate pipelines)

3. Split pipelines strategically

Instead of one monolithic pipeline:

• High-frequency pipeline (incremental tables)
• Low-frequency pipeline (dimension / heavy recompute)

This aligns better with how Lakeflow is designed and reduces friction.

Where the real gap still is

Even beyond UI/YAML convenience, I think the deeper gap is:

No first-class concept of heterogeneous refresh SLAs within a single pipeline DAG

That shows up in multiple ways:

• Can’t express SLA per dataset
• Can’t optimize cost vs freshness tradeoffs cleanly
• Forces either:
  • Over-refreshing everything, or
  • Splitting pipelines (operational overhead)

Bottom line

• You’re correct — not supported natively today
• REST workaround is common but non-ideal
• Splitting pipelines + external config is the least painful current approach
• There’s a clear product gap around per-table refresh policy + scheduling

If this lands natively (especially via DAB), it would significantly improve:

• Cost control
• SLA-driven design
• Pipeline maintainability