Ashwin_DSA
Databricks Employee
Databricks Employee

Hi @IM_01,

It is working as expected. In Lakeflow SDP, streaming tables and materialised views are pipeline‑managed tables, so you can’t change low‑level Delta properties (including delta.columnMapping.mode) with ALTER TABLE … SET TBLPROPERTIES. Those are reserved for the engine, not for direct user DDL.

That also means you can’t enable column mapping + schemaTrackingLocation on an SDP table yourself to work around DELTA_STREAMING_INCOMPATIBLE_SCHEMA_CHANGE_USE_LOG, even though the generic Delta docs talk about that option.

For your scenario:

  • You dropped a column in the source. As explained before, that’s a non‑additive schema change (drop/rename/type change).
  • SDP currently treats that as a breaking change for streaming tables. The supported mitigation is to..
    • Either restore/keep the column in the source and ignore it in your logic, or
    • Run a full refresh of the affected SDP tables so they are rebuilt with the new schema and a fresh checkpoint.

Trying to toggle delta.columnMapping.mode = 'name' on the SDP target table is not supported and won’t unblock this error.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

Regards,
Ashwin | Delivery Solution Architect @ Databricks
Helping you build and scale the Data Intelligence Platform.
***Opinions are my own***