Materialized View with Incremental Refresh - Change Data Feed required on source tables?

[johnb1]

Question:

To create a materialized view with incremental refresh, is the Delta Lake feature Change Data Feed required on all of the source tables that the materialized view consumes? Or, is it not required for the creation of the materialized view, but for running the incremental refresh, i.e. would the materialized view created with the REFRESH POLICY = INCREMENTAL actually only perform full refresh if Change Data Feed is not enabled on at least one the source tables?

This documentation states that Change Data Feed is only recommended (i.e. not required), but I have found other sources stating that it is actually a requirement. I know that cited source is an official one, but I have found mistakes in the official Databricks documentation in the past with other topics, so I don't really trust this information.

[leopedroso1909]

Hi @johnb1 - Change Data Feed (CDF) is actually not an universal requirement for creating an incremental materialized view, but it will help with performance aspects for your pipeline and support on incremental strategy.

By using REFRESH POLICY = INCREMENTAL, Databricks can fall back to a full refresh when incremental refresh is not possible. By contrast, you may also want use REFRESH POLICY = INCREMENTAL STRICT with CDF disabled to test whether the view can be refreshed incrementally without silently falling back to a full refresh.

[data_pulse]

@johnb1 

I tested this on Databricks with multiple runs to verify whether Change Data Feed (CDF) is required on source Delta tables for incremental Materialized View refresh.

I created three source configurations and tested INSERT, UPDATE, and DELETE operations against the same GROUP BY Materialized View query.

Source configuration

INSERT	UPDATE	DELETE
Row tracking ON, CDF OFF	Incremental	Incremental	Incremental
Row tracking ON, CDF ON	Incremental	Incremental	Incremental
Row tracking OFF, CDF OFF	Incremental	Incremental	Incremental

All nine refreshes used the GROUP_AGGREGATE incremental maintenance technique, confirmed through Lakeflow planning events and matched with the expected aggregate results.

An important recent development is Automatic Change Data Feed (Auto CDF). On supported Unity Catalog Delta tables with row tracking enabled, Databricks can compute change data automatically without explicitly setting delta.enableChangeDataFeed=true. This is different from legacy CDF, which requires that property.

Interestingly, my test also showed incremental refresh with both CDF and row tracking disabled for this particular aggregation. So explicit source CDF isn't a universal requirement for incremental MV refresh, although different queries may have additional requirements.

Regarding refresh policies:

• INCREMENTAL prefers incremental processing but allows full-recompute fallback.
• INCREMENTAL STRICT requires incrementalization for normal refreshes, initial population may still use a full computation.

My suggestion: Check the actual refresh technique in Lakeflow's planning events or incrementalization insights rather than assuming the presence or absence of CDF determines refresh behaviour.

These observations are specific to the tested query and runtime, not a guarantee for every MV definition.

[LingeshK]

Hello @johnb1,

Following up with the doc references behind the two answers above, and thank you for the test runs in the second one.

Change data feed is a performance recommendation. No documented check, error code or refresh rule makes it a condition for creating the view or for refreshing it incrementally. Row tracking is the property that some queries need, and the docs mark which clauses need it.

Some pointers about the source tables:

• Change data feed: Enabling change data feed is recommended for better incremental refresh performance. The optimisation section recommends deletion vectors, row tracking and change data feed on every source table "to get the best performance".
• Row tracking: The table of supported clauses stars the ones that need row tracking: SELECT expressions, UNION ALL, WHERE and HAVING, and the joins. GROUP BY has no star. The standalone page words it more strongly and says row tracking must be enabled on Delta source tables. Enabling it on every source covers both readings.
• The failure reasons: A refresh that cannot meet INCREMENTAL STRICT fails with an error whose reference page lists eight reasons. ROW_TRACKING_NOT_ENABLED is one of them. None of the eight mentions change data feed.
• The suggested fixes: When a refresh falls back to a full recompute, the incrementalization insights suggest enabling row tracking or deletion vectors, rewriting an operator, or moving to serverless compute. Change data feed is not on that list.

Q1. "To create a materialized view with incremental refresh, is the Delta Lake feature Change Data Feed required on all of the source tables that the materialized view consumes?"

No. The docs recommend it on every source table for performance, and no requirement names it.

Q2. "Or, is it not required for the creation of the materialized view, but for running the incremental refresh, i.e. would the materialized view created with the REFRESH POLICY = INCREMENTAL actually only perform full refresh if Change Data Feed is not enabled on at least one the source tables?"

No to both parts. Under INCREMENTAL, the refresh runs a full refresh in two cases: Incremental refresh is not available, or the view needs re-initialization, for example after a schema change. For the first case, the refresh policy section gives "row-tracking on a source table is turned off" as the example. No page names a missing change data feed as a cause.

One detail the answers above do not cover: If the query cannot be refreshed incrementally at all, the CREATE fails under INCREMENTAL as well as INCREMENTAL STRICT. A view that you created with INCREMENTAL has passed the query shape check.

How the test above fits the docs: The GROUP BY view refreshed incrementally with row tracking set to off, and GROUP BY has no star in the table. Plain column references and aggregates in a GROUP BY view did not trigger the row tracking check in that test, even though the table also stars SELECT expressions. One case the thread has not covered yet is a starred clause, such as a WHERE filter, on a source without row tracking. For that case the docs predict a failure under INCREMENTAL STRICT, with ROW_TRACKING_NOT_ENABLED as the reason. Automatic upgrades can turn row tracking on for Unity Catalog managed tables, so the sketch confirms the property before it creates the view. Either result would be useful here:

-- Source without row tracking or change data feed
CREATE TABLE main.sandbox.rt_test_src (region STRING, amount BIGINT)
TBLPROPERTIES (delta.enableRowTracking = false, delta.enableChangeDataFeed = false);

INSERT INTO main.sandbox.rt_test_src VALUES ('EMEA', 10), ('AMER', 20);

-- Confirm row tracking is still off before you go on
SHOW TBLPROPERTIES main.sandbox.rt_test_src ('delta.enableRowTracking');

-- A starred clause (WHERE), with no fallback allowed
CREATE MATERIALIZED VIEW main.sandbox.rt_test_mv
REFRESH POLICY INCREMENTAL STRICT
AS SELECT region, amount FROM main.sandbox.rt_test_src WHERE amount > 5;

-- If the CREATE succeeds, change the source and refresh
INSERT INTO main.sandbox.rt_test_src VALUES ('EMEA', 15);
REFRESH MATERIALIZED VIEW main.sandbox.rt_test_mv;

Before you rely on this:

• Testing: I have not run this sketch. The GROUP BY results above are the tested evidence in this thread.
• The cost model: Under the default AUTO policy, an eligible view can still get a full recompute when the cost model judges it cheaper. A full refresh under AUTO does not prove that a requirement is missing.
• Serverless: Incremental refresh runs on serverless compute. Views in pipelines that do not use serverless are always fully recomputed.
• Recreated sources: Re-enable row tracking whenever you recreate a source table.
• Enabling row tracking on a live table: It assigns IDs to every existing row, which can take a long time, and concurrent writes fail while it runs. Pause streaming writes first.
• Where "change data feed is required" may come from: Reading a change data feed from a materialized view is a separate Beta feature with its own requirements. It covers changes that flow out of the view to downstream readers.

Reference documentation:

• Incremental refresh for materialized views: Covers the supported clauses and their row tracking star, the three recommended properties, refresh policies and the event log query. It does not say what change data feed changes inside a refresh.
• Use standalone materialized views: States that incremental refresh needs row tracking and that change data feed is recommended for performance.
• REFRESH POLICY clause: Covers what each policy does at create and at refresh, and the error a strict refresh returns.
• MATERIALIZED_VIEW_NOT_INCREMENTALIZABLE error class: Lists eight reasons a strict refresh can fail. None is about change data feed.
• Automatic upgrades: Lists the features, row tracking included, that Databricks turns on for Unity Catalog managed tables, and how to spot them in the table history.
• EXPLAIN CREATE MATERIALIZED VIEW: Reports whether a query is eligible before you create the view. The cost model can still choose a full recompute at runtime.

Happy to look at the result if you run the sketch and post it here.

[indradbxlearn]

@johnb1 - Hello Johnb1 -  CDF is not required, either to create the materialized view or to refresh it incrementally. The current Databricks docs (updated October 2026) list it only as a recommendation for better performance.

Row tracking is what can gate incremental refresh. Operators like joins, UNION ALL, WHERE/HAVING and SELECT expressions need it on the source tables, while a plain GROUP BY does not.

Your second reading is closer to how it works:

• CREATE ... REFRESH POLICY INCREMENTAL always succeeds, because the policy is a preference, not a guarantee.
• At refresh time, INCREMENTAL falls back to a full recompute if the query plan can’t be incrementalized, for example when row tracking is missing on a source.
• Fallback is per query operator, not per table feature, so a missing property only hurts if your query needs it.
• Incremental refresh only runs on serverless compute. Classic compute always does a full recompute.

To verify it yourself:

1. Run EXPLAIN CREATE MATERIALIZED VIEW ... to check whether the query is incrementalizable.
2. Use REFRESH POLICY INCREMENTAL STRICT, which fails instead of silently doing a full refresh. Test it with CDF off and row tracking on.
3. Query the pipeline event log (event_type = 'planning_information') to see whether a refresh was FULL_RECOMPUTE or incremental.

Practical takeaway: Enable row tracking and deletion vectors on all source tables, and treat CDF as optional. Older docs and blog posts did describe CDF as a prerequisite, which likely explains the conflicting sources you found.