DLT Apply Changes problem

[_singh_vish]

Hi All, I am working on DLT pipeline, to create SCD2 for my bronze layer, my architecture has 4 layers, namely Raw, Bronze, Silver, Gold. I am ingesting data directly into raw, and then I am creating history(SCD2) into bronze. 

My code:

    @Dlt.view(name=source_name)
    def source():
        return (
            spark.readStream.format("delta")
            .option("readChangeData", "true")
            .option("startingVersion", 0) 
            .option("mergeSchema", "true")
            .table(source_table_location)
            .select(
                *[col(source_col).alias(target_col) for source_col, target_col in alias_config.items()]
            )
        )

    # Defining the target table for SCD2
    dlt.create_streaming_table(
        name=target_table_name
    )

    dlt.apply_changes(
        target=target_table_name,
        source=source_name,
        keys=keys,
        sequence_by=col(last_updated_column),
        track_history_except_column_list=track_history_except_column_list,
        stored_as_scd_type="2",
        apply_as_deletes=col(deleted_flag) == True if deleted_flag else False
    )  

when the run the pipeline, it shows DLT does not support schema evolution, and other similar errors related to schema being changed somehow.
Is there a way to resolve this?
Also, even though the data is in delta table as batch upload, but it has last modified date available.

Can someone help?

DELTA_SCHEMA_CHANGED_WITH_STARTING_OPTIONS(The error code)

[Stefan-Koch]

Hi @_singh_vish 

Can you provide some error-logs/messages?

[_singh_vish]

I cannot send the exact error log, but here's a high level of that:
[DELTA_SCHEMA_CHANGED_WITH_STARTING_OPTIONS] Detected schema change in version 6:\nstreaming source schema: root\n

I want to know, how can i handle tables, which has a last modified date, currently i am using dlt.apply_changes function for this, where i am forcing my delta table as a streaming source and then writing in a streaming table as my target. But this throws errors.

Is there any other way to do this? If yes, then please tell. Thanks!

[_singh_vish]

This is the table history, and it breaks on version 6, and also on version 8, 
I have not checked other versions after 8, but will expect a similar result for those too.