- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-22-2026 01:38 PM
We have Auto-TTL configured on a Lakeflow pipeline streaming table via create_streaming_table(auto_ttl={"timestamp_column": "_discontinued_on", "expire_in_days": 77}). The autottl.timestampColumn and autottl.expireInDays properties are correctly set on the table. The table is a Unity Catalog managed streaming table (is_managed_location: true), serverless, ADVANCED edition, CURRENT channel, production mode.
The problem: Auto-TTL has never fired. 1.5M rows are past the 77-day expiry, with the oldest _discontinued_on dating back to 2021. Zero rows in system.storage.predictive_optimization_operations_history for this table.
Root cause we've identified: The pipeline runtime stamps predictive_optimization: DISABLE on the table at creation and on every update. The Auto-TTL docs state: "Turning off predictive optimization on a table with auto time-to-live enabled prevents auto time-to-live from running." So the pipeline's own behavior blocks the feature.
What we've ruled out:
- PO is enabled at account, metastore, catalog, and schema level (all show ENABLE inherited from METASTORE)
- PO works on regular managed tables in the same catalog — we tested with a plain managed table with Auto-TTL and it successfully ran DELETE + PURGE within hours
- We checked all 218 streaming tables across the catalog — zero PO operations on any of them. Every one shows predictive_optimization: DISABLE
- ALTER TABLE ... ENABLE PREDICTIVE OPTIMIZATION fails because the pipeline owns the table
- No environment version is set on the pipeline (environment_version: None), but the docs don't tie environment version to PO enablement
- Using dlt.create_streaming_table vs @DP.table doesn't matter — the autottl properties are set correctly either way
- pipelines.autoOptimize.managed controls the pipeline's own OPTIMIZE/VACUUM scheduling, but the docs only say "for pipelines managed by predictive optimization, this property is not used" — they don't say setting it to false causes PO to take over
The documentation contradiction:
- Auto-TTL docs say it's available for "Streaming tables with Lakeflow pipelines"
- Auto-TTL requirements say "You must turn on predictive optimization" and "turning off predictive optimization prevents auto time-to-live from running"
- The Lakeflow pipelines docs say "Pipelines perform automatic maintenance using predictive optimization"
- But the pipeline runtime stamps predictive_optimization: DISABLE on every streaming table it manages
- You cannot ALTER a pipeline-owned streaming table to enable PO
- There is no documented pipeline configuration to enable PO on the table
So Auto-TTL is documented to work on pipeline streaming tables, but the prerequisite (PO enabled on the table) is impossible to satisfy through any documented mechanism.
Environment:
- Azure Databricks, Premium plan
- Serverless pipelines, ADVANCED edition, CURRENT channel
- Unity Catalog, managed storage
- Pipeline uses create_auto_cdc_flow (SCD Type 1 from silver change feed)
- PO rollout note says "expected to complete by August 2026" — we're past that date, and PO works on non-pipeline managed tables but not on any pipeline streaming tables
Questions for the community:
- Has anyone gotten Auto-TTL to work on a pipeline-managed streaming table? If so, what was the configuration?
- Is there a pipeline-level configuration or feature flag that transitions pipeline tables from pipeline-managed maintenance to PO-managed maintenance?
- Is the PO rollout for pipeline streaming tables on a separate timeline from the general PO rollout for managed tables?
- For anyone who has filed a support case on this — what was the resolution?
Any insight appreciated. We have a support case planned but wanted to see if anyone in the community has navigated this already.