Hello @cdn_yyz_yul, Thanksto @Jen_Evangelista for the pipeline-config versus table-property framing and to @data_pulse for the Free Edition repro.
On the property question: I can't point to a doc line, but the evidence says it's expected that pipelines.externalMetadata.enabled doesn't show in SHOW TBLPROPERTIES. data_pulse never saw it in any of the three configuration paths, and in your case the metadata log did get written (REPAIR couldn't have produced versions 3 through 35 otherwise). The flag is consumed by the pipeline, not stored on the table. Note that data_pulse's failure is at an earlier step than yours: their log never got created at all.
There is a clue: Your dump has these two lines:
delta.enableChangeDataFeed,true
delta.feature.changeDataFeed,supported
That's legacy change data feed, turned on. The MV docs say materialized views use automatic CDF and you don't turn on the change feed itself. The automatic CDF docs say the two can't be used together, and the migration step is to unset delta.enableChangeDataFeed. With legacy on, the reader goes looking for materialized change records at each version, the pipeline never wrote any, and you get "change data was not recorded" at every version, before and after REPAIR. It fits all the facts. I don't think you set it; my understanding is pipelines turn legacy CDF on by default for the tables they manage. The experiment:
- Add
"delta.enableChangeDataFeed": "false" to the MV's table_properties and run an update.
- Check
SHOW TBLPROPERTIES my_catalog.my_schema.my_mv ('delta.enableChangeDataFeed'). If the pipeline put it back, that goes straight in the ticket.
- If it stuck, run
REPAIR TABLE ... SYNC METADATA once more and read from the newest version with an explicit version number.
Caveat: if a downstream MV refreshes incrementally off this one, turning CDF off may push it to full recompute. Fine for a terminal gold MV in a test pipeline.
On the plain-table test, two things. First, automatic CDF needs row tracking on the table, and nothing in your history output shows whether that test table had it. The clean test is a fresh table with delta.enableRowTracking = true and no delta.enableChangeDataFeed, two writes, then table_changes from version 1 on DBR 19. If that works, automatic CDF is fine in your workspace and the MV problem is isolated to the legacy property. If it fails, automatic CDF itself isn't live for you, and that's the headline of the ticket instead. Second, a period with no changes should return zero rows, not DELTA_MISSING_CHANGE_DATA, so the error was never "nothing changed."
On streaming from the MV: fair hit. The docs do list a streaming read with readChangeFeed=true as supported for this beta, from a different PREVIEW pipeline or Databricks SQL, but it rides the same path as the batch read. Until table_changes() works, the stream won't either. Keep that exact error for the report.
If the experiment doesn't get you there, I'd treat this as a Beta product issue rather than a configuration problem, and you've got a crisp case: MV name, pipeline ID, runtime and compute type, the full property dump, confirmation REPAIR succeeded, the version range (3 through 35), a failing explicit-version read, the row-tracked plain-table result, and this thread.
References:
Regards, Louis.
