anuj_lathi
Databricks Employee
Databricks Employee

Short version: your diagnosis holds up. As far as I know, there is no pipeline setting, table property, or flag that flips a pipeline-owned streaming table to PO-managed maintenance. I wouldn't expect Auto-TTL to run on a table that is stamped DISABLE and can't be altered, so this is a support-case/product-gap issue rather than a config mistake on your side.

Why the usual knobs don't help

  • Pipeline settings (edition, channel, serverless, configuration, etc.) and pipelines.* Spark properties only control pipeline runtime behavior [1]. They can be set pipeline-wide in the configuration object or per dataset via spark_conf [1]. None of the properties I know of in that family change the table's PO state.
  • pipelines.autoOptimize.managed governs the pipeline's own maintenance. Setting it to false just turns off the pipeline's OPTIMIZE/VACUUM. It doesn't hand the table to PO, and you'd end up with neither maintenance path.
  • ALTER TABLE ... ENABLE PREDICTIVE OPTIMIZATION fails because the pipeline owns the table's lifecycle. That's consistent with what you're seeing across all 218 tables.
  • Your control test (a plain managed table running DELETE + PURGE) shows the account, metastore, and catalog setup is fine. The variable is the pipeline-owned table.

Your questions

  1. Working config on a pipeline streaming table: I haven't seen one, and I don't have a configuration to give you.
  2. Pipeline-level flag to move to PO-managed maintenance: none that I'm aware of. I wouldn't spend more time hunting through pipeline properties.
  3. Separate rollout timeline: that's plausible given what you're observing, but only Databricks can confirm it. Put it as an explicit question in the case.
  4. Support case resolutions: I haven't seen one posted.

What to put in the support case

  • Pipeline ID, table full name, and the output of DESCRIBE EXTENDED / SHOW TBLPROPERTIES showing autottl.* set and predictive_optimization: DISABLE.
  • Your control test: the same catalog, with Auto-TTL running on a regular managed table.
  • The 218-table check showing zero PO operations on any streaming table.
  • Three direct questions:
    • Is Auto-TTL supported on pipeline-owned streaming tables today?
    • If so, what enables PO on them?
    • If not, should the docs be corrected?

Workarounds in the meantime

You have 1.5M expired rows, so I wouldn't wait on this.

  1. Handle expiry inside the pipeline. Since you're running an AUTO CDC flow (SCD Type 1) from a silver change feed, emit delete events for expired keys from the source side and use the flow's delete handling (apply_as_deletes). The retention logic then lives in the pipeline, and the table stays consistent with its flow.
  2. Scheduled DELETE as the table owner. DML on streaming tables is supported in some scenarios, so a scheduled job running DELETE FROM ... WHERE _discontinued_on < current_date() - INTERVAL 77 DAYS is an option. Test it first. Check how it interacts with your AUTO CDC flow and any downstream streaming readers (they'd need to tolerate non-append commits, e.g. skipChangeCommits). Also confirm that the pipeline's own OPTIMIZE/VACUUM eventually reclaims the files.
  3. Different table type. If you don't need the streaming-table semantics on this particular target, a regular managed Delta table written by a job gets you working Auto-TTL today, as your control test shows.

I'd go with option 1 if you can, and treat the support case as the route to a real Auto-TTL fix.

References

[1] Pipeline properties reference | Databricks on AWS — https://docs.databricks.com/aws/en/ldp/properties

Anuj Lathi
Solutions Engineer @ Databricks

View solution in original post