Auto CDC fLow without CDF?

[kevinzhang29]

Auto CDC flow works with source table CDF enabled, but fails when CDF is disabled.

The source table is updated via INSERT OVERWRITE.

IS CDF mandatory?

[amirabedhiafi]

Hello Kevin !

It happened with me once and that's how I understood that for auto cdc the delta source needs a change feed.

When it is not available you should use AUTO CDC FROM SNAPSHOT which compares snapshots and with do the synth of the changes instead.

And that also explains why you will have a failure when the source is updated with INSERT OVERWRITE and CDF is off. 

[DivyaandData]

Yes, @kevinzhang29 . For Auto CDC with a Delta source table, a change data feed (CDF) (i.e., a CDC feed) is required. AUTO CDC is explicitly designed to read from a CDC/change feed source such as Delta CDF, not from plain snapshots.

When you don’t have a change feed (CDF off, or an upstream system that only gives you full table dumps / INSERT OVERWRITE), you should switch to AUTO CDC FROM SNAPSHOT instead. That API compares consecutive snapshots, infers inserts/updates/deletes for you, and then runs the same SCD1/SCD2 logic on top of that synthetic change feed.

That’s why your flow:

• Works when CDF is enabled on the Delta source (AUTO CDC can see row-level changes).
• Fails when CDF is disabled and the source is updated via INSERT OVERWRITE (there is no CDC feed for AUTO CDC to consume; it’s just seeing full table rewrites).

So in short:

• Yes, CDF is mandatory for AUTO CDC over a Delta source.
• If you can’t enable CDF, use AUTO CDC FROM SNAPSHOT and point it at the snapshot table/view that’s being overwritten.

If this resolves your question, could you please mark it as “Accept as Solution”? It helps other users quickly find the right fix.