Hi everyone,

I am implementing SCD Type 2 in the Bronze layer of a Delta Live Table (DLT) pipeline using Structured Streaming. I am curious about the necessity of having a table or view before loading data into the Bronze table. Without this, it seems like change tracking (e.g., Change Data Feed) doesn't start until the Silver table.

Here’s an example of code where the Change Data Feed is enabled, but it begins only at the Silver table (the next layer):

@Dlt.table(
comment="New customer data ingested from cloud landing source",
table_properties={
"delta.enableChangeDataFeed": "true", 
"quality": "bronze"
}
)
def Customer_Bronze():
df = (spark.readStream.format("cloudFiles")
.option("cloudFiles.format", "json")
.option("inferSchema", True)
.option("cloudFiles.inferColumnTypes", "true")
.option("recursiveFileLookup", "true")
.load(landing_json_path) # Loaded from the raw storage location.
)
return df

What are the best practices for setting up the Bronze table in this case? Should I rely on staging tables or views for proper ingestion and change tracking? Any advice on optimizing SCD Type 2 with DLT and Structured Streaming would be appreciated!

Thank you!