Hi @Islam_hoti,

This is a central design decision for any production-grade Lakehouse. In my experience architecting petabyte-scale environments, the goal is to minimize "Pipeline Friction" without sacrificing "Data Governance."

Here is the "Architectural Standard" I recommend for this scenario:

1. The Bronze Strategy: "Capture Everything"

In the Bronze layer, your primary goal is to ensure that no data is lost due to upstream changes.

• Mode: I recommend staying with addNewColumns but always combining it with rescuedDataColumn.
• Why: rescuedDataColumn acts as your "safety net." If a field changes in a way that addNewColumns can’t handle (like a data type conflict), the data isn't lost; it’s captured in a JSON blob for later recovery.
• The Restart: The UnknownFieldException and subsequent restart are actually a "feature," not a bug. It ensures the state/checkpoint is updated with the new metadata. In a Lakeflow Job, configuring an automatic retry makes this process transparent and self-healing.

1. The Silver Strategy: "Enforce and Cleanse"

This is where you solve the "Silent Schema Change" problem. While Bronze is flexible, Silver should be strict.

• We allow Bronze to evolve automatically to keep the lights on.
• In the Silver transformation, we use Schema Enforcement. If the new column in Bronze isn't explicitly handled in your Silver logic, it simply stays in Bronze until an engineer or architect decides how to map it. This prevents "junk" columns from polluting your business-ready tables.

1. The Modern Twist: Type Widening & Unity Catalog

If you are on a recent Databricks Runtime (13.3+), you should definitely evaluate Type Widening. It allows Auto Loader to evolve a column from an INT to a BIGINT without failing the stream or requiring a rewrite of the Delta table. Under Unity Catalog, this is managed seamlessly, ensuring that your governance and lineage remain intact even as the schema grows.

For a production environment, I prefer Option 1 (Auto-restart) + Option 2 (Rescue Mode). Enforcing the schema at the ingestion layer (Bronze) is a recipe for operational headaches—upstream teams will inevitably break your pipeline. By capturing the change in Bronze and enforcing the logic in Silver, you maintain high availability while keeping strict governance where it matters most.