Auto-TTL on pipeline streaming table not firing

[monson_mark]

We have Auto-TTL configured on a Lakeflow pipeline streaming table via create_streaming_table(auto_ttl={"timestamp_column": "_discontinued_on", "expire_in_days": 77}). The autottl.timestampColumn and autottl.expireInDays properties are correctly set on the table. The table is a Unity Catalog managed streaming table (is_managed_location: true), serverless, ADVANCED edition, CURRENT channel, production mode.

The problem: Auto-TTL has never fired. 1.5M rows are past the 77-day expiry, with the oldest _discontinued_on dating back to 2021. Zero rows in system.storage.predictive_optimization_operations_history for this table.

Root cause we've identified: The pipeline runtime stamps predictive_optimization: DISABLE on the table at creation and on every update. The Auto-TTL docs state: "Turning off predictive optimization on a table with auto time-to-live enabled prevents auto time-to-live from running." So the pipeline's own behavior blocks the feature.

What we've ruled out:

• PO is enabled at account, metastore, catalog, and schema level (all show ENABLE inherited from METASTORE)
• PO works on regular managed tables in the same catalog — we tested with a plain managed table with Auto-TTL and it successfully ran DELETE + PURGE within hours
• We checked all 218 streaming tables across the catalog — zero PO operations on any of them. Every one shows predictive_optimization: DISABLE
• ALTER TABLE ... ENABLE PREDICTIVE OPTIMIZATION fails because the pipeline owns the table
• No environment version is set on the pipeline (environment_version: None), but the docs don't tie environment version to PO enablement
• Using dlt.create_streaming_table vs @DP.table doesn't matter — the autottl properties are set correctly either way
• pipelines.autoOptimize.managed controls the pipeline's own OPTIMIZE/VACUUM scheduling, but the docs only say "for pipelines managed by predictive optimization, this property is not used" — they don't say setting it to false causes PO to take over

The documentation contradiction:

1. Auto-TTL docs say it's available for "Streaming tables with Lakeflow pipelines"
2. Auto-TTL requirements say "You must turn on predictive optimization" and "turning off predictive optimization prevents auto time-to-live from running"
3. The Lakeflow pipelines docs say "Pipelines perform automatic maintenance using predictive optimization"
4. But the pipeline runtime stamps predictive_optimization: DISABLE on every streaming table it manages
5. You cannot ALTER a pipeline-owned streaming table to enable PO
6. There is no documented pipeline configuration to enable PO on the table

So Auto-TTL is documented to work on pipeline streaming tables, but the prerequisite (PO enabled on the table) is impossible to satisfy through any documented mechanism.

Environment:

• Azure Databricks, Premium plan
• Serverless pipelines, ADVANCED edition, CURRENT channel
• Unity Catalog, managed storage
• Pipeline uses create_auto_cdc_flow (SCD Type 1 from silver change feed)
• PO rollout note says "expected to complete by August 2026" — we're past that date, and PO works on non-pipeline managed tables but not on any pipeline streaming tables

Questions for the community:

1. Has anyone gotten Auto-TTL to work on a pipeline-managed streaming table? If so, what was the configuration?
2. Is there a pipeline-level configuration or feature flag that transitions pipeline tables from pipeline-managed maintenance to PO-managed maintenance?
3. Is the PO rollout for pipeline streaming tables on a separate timeline from the general PO rollout for managed tables?
4. For anyone who has filed a support case on this — what was the resolution?

Any insight appreciated. We have a support case planned but wanted to see if anyone in the community has navigated this already.

[ThomazNeto]

Hi,

You did the homework, so I'll skip repeating it and add three things.

First, the docs contradiction you found is real, and it's worse than you listed. The GDPR page says "The recommended way is Predictive optimization for Unity Catalog managed tables, which intelligently maintains both streaming tables and materialized views", and in the same breath "For tables without predictive optimization enabled, Lakeflow pipelines automatically perform maintenance tasks within 24 hours". Meanwhile the PO page never mentions pipelines at all. And in another thread here two days ago a Databricks employee stated the exact opposite of what you observe: that PO runs on pipeline-published tables even when it's disabled at catalog or schema level, and that there is no way to turn it off. Your 218 tables with zero PO operations say otherwise. Worth pasting that thread into your support case, because Databricks is currently telling customers two incompatible stories.
https://docs.databricks.com/aws/en/ldp/gdpr
https://docs.databricks.com/aws/en/optimizations/predictive-optimization
https://community.databricks.com/t5/data-engineering/predictive-optimization-for-streaming-tables-in...

Second, two checks that will strengthen the case. On DBR 18 LTS or above, DESCRIBE TABLE EXTENDED your_table AS JSON returns a predictive_optimization_evaluations field with the reasons PO skipped a table; the docs say results can take 24 hours to appear. If that field is empty for a pipeline table while it's populated for your plain managed table, that's hard evidence PO never evaluates them. Also run DESCRIBE HISTORY on the streaming table and look for DELETE operations; the Auto TTL page names that as the second way to verify, independent of the system table.
https://docs.databricks.com/aws/en/tables/operations/auto-ttl

Third, the workaround while support sorts it out. Streaming tables accept DML from a SQL warehouse or a standard-access cluster on 13.3+, so a scheduled job running DELETE FROM your_table WHERE _discontinued_on < current_date() - INTERVAL 77 DAYS gives you the same result today. Two caveats straight from the docs: any downstream stream reading that table needs skipChangeCommits or it fails on the delete (Auto TTL has the same requirement), and you'll want the pipeline to not full-refresh the table or the deletes come back. It's ugly compared to a one-line auto_ttl, but 1.5 million expired rows is a compliance problem now, not after the ticket closes.
https://docs.databricks.com/aws/en/ldp/unity-catalog

Please post what support says. This affects everyone using Auto TTL on pipeline tables and there's nothing in the docs about it yet.

[anuj_lathi]

Short version: your diagnosis holds up. As far as I know, there is no pipeline setting, table property, or flag that flips a pipeline-owned streaming table to PO-managed maintenance. I wouldn't expect Auto-TTL to run on a table that is stamped DISABLE and can't be altered, so this is a support-case/product-gap issue rather than a config mistake on your side.

Why the usual knobs don't help

• Pipeline settings (edition, channel, serverless, configuration, etc.) and pipelines.* Spark properties only control pipeline runtime behavior [1]. They can be set pipeline-wide in the configuration object or per dataset via spark_conf [1]. None of the properties I know of in that family change the table's PO state.
• pipelines.autoOptimize.managed governs the pipeline's own maintenance. Setting it to false just turns off the pipeline's OPTIMIZE/VACUUM. It doesn't hand the table to PO, and you'd end up with neither maintenance path.
• ALTER TABLE ... ENABLE PREDICTIVE OPTIMIZATION fails because the pipeline owns the table's lifecycle. That's consistent with what you're seeing across all 218 tables.
• Your control test (a plain managed table running DELETE + PURGE) shows the account, metastore, and catalog setup is fine. The variable is the pipeline-owned table.

Your questions

1. Working config on a pipeline streaming table: I haven't seen one, and I don't have a configuration to give you.
2. Pipeline-level flag to move to PO-managed maintenance: none that I'm aware of. I wouldn't spend more time hunting through pipeline properties.
3. Separate rollout timeline: that's plausible given what you're observing, but only Databricks can confirm it. Put it as an explicit question in the case.
4. Support case resolutions: I haven't seen one posted.

What to put in the support case

• Pipeline ID, table full name, and the output of DESCRIBE EXTENDED / SHOW TBLPROPERTIES showing autottl.* set and predictive_optimization: DISABLE.
• Your control test: the same catalog, with Auto-TTL running on a regular managed table.
• The 218-table check showing zero PO operations on any streaming table.
• Three direct questions:
  • Is Auto-TTL supported on pipeline-owned streaming tables today?
  • If so, what enables PO on them?
  • If not, should the docs be corrected?

Workarounds in the meantime

You have 1.5M expired rows, so I wouldn't wait on this.

1. Handle expiry inside the pipeline. Since you're running an AUTO CDC flow (SCD Type 1) from a silver change feed, emit delete events for expired keys from the source side and use the flow's delete handling (apply_as_deletes). The retention logic then lives in the pipeline, and the table stays consistent with its flow.
2. Scheduled DELETE as the table owner. DML on streaming tables is supported in some scenarios, so a scheduled job running DELETE FROM ... WHERE _discontinued_on < current_date() - INTERVAL 77 DAYS is an option. Test it first. Check how it interacts with your AUTO CDC flow and any downstream streaming readers (they'd need to tolerate non-append commits, e.g. skipChangeCommits). Also confirm that the pipeline's own OPTIMIZE/VACUUM eventually reclaims the files.
3. Different table type. If you don't need the streaming-table semantics on this particular target, a regular managed Delta table written by a job gets you working Auto-TTL today, as your control test shows.

I'd go with option 1 if you can, and treat the support case as the route to a real Auto-TTL fix.

References

[1] Pipeline properties reference | Databricks on AWS — https://docs.databricks.com/aws/en/ldp/properties