Auto-TTL on pipeline streaming table not firing

monson_mark
New Contributor II

We have Auto-TTL configured on a Lakeflow pipeline streaming table via create_streaming_table(auto_ttl={"timestamp_column": "_discontinued_on", "expire_in_days": 77}). The autottl.timestampColumn and autottl.expireInDays properties are correctly set on the table. The table is a Unity Catalog managed streaming table (is_managed_location: true), serverless, ADVANCED edition, CURRENT channel, production mode.

The problem: Auto-TTL has never fired. 1.5M rows are past the 77-day expiry, with the oldest _discontinued_on dating back to 2021. Zero rows in system.storage.predictive_optimization_operations_history for this table.

Root cause we've identified: The pipeline runtime stamps predictive_optimization: DISABLE on the table at creation and on every update. The Auto-TTL docs state: "Turning off predictive optimization on a table with auto time-to-live enabled prevents auto time-to-live from running." So the pipeline's own behavior blocks the feature.

What we've ruled out:

  • PO is enabled at account, metastore, catalog, and schema level (all show ENABLE inherited from METASTORE)
  • PO works on regular managed tables in the same catalog — we tested with a plain managed table with Auto-TTL and it successfully ran DELETE + PURGE within hours
  • We checked all 218 streaming tables across the catalog — zero PO operations on any of them. Every one shows predictive_optimization: DISABLE
  • ALTER TABLE ... ENABLE PREDICTIVE OPTIMIZATION fails because the pipeline owns the table
  • No environment version is set on the pipeline (environment_version: None), but the docs don't tie environment version to PO enablement
  • Using dlt.create_streaming_table vs @DP.table doesn't matter — the autottl properties are set correctly either way
  • pipelines.autoOptimize.managed controls the pipeline's own OPTIMIZE/VACUUM scheduling, but the docs only say "for pipelines managed by predictive optimization, this property is not used" — they don't say setting it to false causes PO to take over

The documentation contradiction:

  1. Auto-TTL docs say it's available for "Streaming tables with Lakeflow pipelines"
  2. Auto-TTL requirements say "You must turn on predictive optimization" and "turning off predictive optimization prevents auto time-to-live from running"
  3. The Lakeflow pipelines docs say "Pipelines perform automatic maintenance using predictive optimization"
  4. But the pipeline runtime stamps predictive_optimization: DISABLE on every streaming table it manages
  5. You cannot ALTER a pipeline-owned streaming table to enable PO
  6. There is no documented pipeline configuration to enable PO on the table

So Auto-TTL is documented to work on pipeline streaming tables, but the prerequisite (PO enabled on the table) is impossible to satisfy through any documented mechanism.

Environment:

  • Azure Databricks, Premium plan
  • Serverless pipelines, ADVANCED edition, CURRENT channel
  • Unity Catalog, managed storage
  • Pipeline uses create_auto_cdc_flow (SCD Type 1 from silver change feed)
  • PO rollout note says "expected to complete by August 2026" — we're past that date, and PO works on non-pipeline managed tables but not on any pipeline streaming tables

Questions for the community:

  1. Has anyone gotten Auto-TTL to work on a pipeline-managed streaming table? If so, what was the configuration?
  2. Is there a pipeline-level configuration or feature flag that transitions pipeline tables from pipeline-managed maintenance to PO-managed maintenance?
  3. Is the PO rollout for pipeline streaming tables on a separate timeline from the general PO rollout for managed tables?
  4. For anyone who has filed a support case on this — what was the resolution?

Any insight appreciated. We have a support case planned but wanted to see if anyone in the community has navigated this already.