Louis_Frolio
Databricks Employee
Databricks Employee

Monday

Hello @cdn_yyz_yul, I took a look at both internal and external documentation and here is what I found.

Your setup checklist matches the docs. Row tracking on, pipelines.externalMetadata.enabled set (pipeline and table level, the table property wins), PREVIEW channel, recent runtime. For MV change feeds delta.enableChangeDataFeed isn't the switch anyway, it runs on automatic CDF, so you haven't missed a flag.

Look at where the error points: .../_external_metadata/_delta_log. The MV change feed doesn't read the MV's own Delta log, it reads a separate external metadata log the pipeline writes when that flag is on. Your error says that log doesn't exist yet, so table_changes() can't build version 0. Configured, but never generated (or the update that should've generated it didn't stick).

Here's the order I'd try things in:

  1. Force the metadata to generate. Run this from serverless or a standard cluster on DBR 17.3 or above, as a principal with MODIFY on the MV:

    REPAIR TABLE <catalog>.<schema>.my_mv SYNC METADATA;

  2. Retry with the three-part name, and don't assume version 0 exists. Since you dropped and recreated the MV, the earliest available version may be later than 0. Try a timestamp from just before the full refresh, or set spark.databricks.delta.changeDataFeed.timestampOutOfRange.enabled = true so an out-of-range start returns empty instead of erroring.

  3. Confirm the full refresh actually finished green in the pipeline UI. A failed or partial update won't write the external metadata.

  4. Check for catalog commits. Catalog commits (Beta) isn't compatible with external metadata and has to be disabled first. Run DESCRIBE DETAIL <catalog>.<schema>.my_mv and look at the table features.

  5. Check the metastore-level "external data access" toggle in the account console. The MV CDF docs only mention the dataset-level flag, so I'm not certain it's required here, but if it's off I'd flip it and rerun the pipeline to rule it out.

  6. Rule out the reader. The MV CDF page says DBR 18 LTS or above, the general automatic CDF page says DBR 19 or above. Serverless environment 5 on Spark 4.2 should clear both, but if nothing above moves the needle, try the same query from a serverless SQL warehouse.

Two things to know as you build this out. You can't read an MV's change feed from inside the pipeline that creates it; use a separate PREVIEW pipeline for that pattern. And for pipeline-managed MVs, the supported reset is a full refresh or removing the MV from the definition, not a direct DROP. It worked here, but my not work in other situations.

References:

Let us know how it goes.

Regards, Louis.

0 Kudos