- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-27-2025 08:46 AM
hey @excavator-matt
Let's remember that the Bronze layer is for mere raw ingestion; this provides a baseline for auditing and to start applying transformations based on the different use cases you need to serve. Systems and their requirements change over time, which leads to the addition or removal of columns, schema evolution, and is typically applied to a table. The full practice can also be extended to multiple tables, depending on your scenario: When creating/refreshing your tables using the "SELECT * from ..." from your source tables with Schema Evolution, but at some point in your flow, you'll need to establish a schema (Schema Enforcement), as you need to guarantee reliability to your consumers.
LDP handles incremental ingestion, so your bronze layer serves as the landing point for your newly arriving data. You receive new/updated/deleted rows, which are normally assigned to an "operation" column that determines the action to apply to each row. From here, we apply a change data feed from your Bronze to your Silver table. This will synchronize (via upserts) the new data arriving with your already transformed baseline data. In the guide, there are two steps: the first step is to create the baseline data, and the second step 2 to apply the CDC with new upcoming data.
My suggestion is to "Split and conquer" all these steps, once you have it working, shape it while cutting corners while keeping core functionality. First ingest all data, prepare, transform, then synchronize with your baseline(cdc).
Thank you, I hope you find this helpful!