DLT Apply Changes problem
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-01-2025 10:04 AM
Hi All, I am working on DLT pipeline, to create SCD2 for my bronze layer, my architecture has 4 layers, namely Raw, Bronze, Silver, Gold. I am ingesting data directly into raw, and then I am creating history(SCD2) into bronze.
My code:
@Dlt.view(name=source_name)
def source():
return (
spark.readStream.format("delta")
.option("readChangeData", "true")
.option("startingVersion", 0)
.option("mergeSchema", "true")
.table(source_table_location)
.select(
*[col(source_col).alias(target_col) for source_col, target_col in alias_config.items()]
)
)
# Defining the target table for SCD2
dlt.create_streaming_table(
name=target_table_name
)
dlt.apply_changes(
target=target_table_name,
source=source_name,
keys=keys,
sequence_by=col(last_updated_column),
track_history_except_column_list=track_history_except_column_list,
stored_as_scd_type="2",
apply_as_deletes=col(deleted_flag) == True if deleted_flag else False
)
when the run the pipeline, it shows DLT does not support schema evolution, and other similar errors related to schema being changed somehow.
Is there a way to resolve this?
Also, even though the data is in delta table as batch upload, but it has last modified date available.
Can someone help?
DELTA_SCHEMA_CHANGED_WITH_STARTING_OPTIONS(The error code)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-01-2025 09:03 PM
Hi @_singh_vish
Can you provide some error-logs/messages?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-02-2025 01:13 AM
I cannot send the exact error log, but here's a high level of that:
[DELTA_SCHEMA_CHANGED_WITH_STARTING_OPTIONS] Detected schema change in version 6:\nstreaming source schema: root\n
I want to know, how can i handle tables, which has a last modified date, currently i am using dlt.apply_changes function for this, where i am forcing my delta table as a streaming source and then writing in a streaming table as my target. But this throws errors.
Is there any other way to do this? If yes, then please tell. Thanks!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-02-2025 01:51 AM
This is the table history, and it breaks on version 6, and also on version 8,
I have not checked other versions after 8, but will expect a similar result for those too.