Hi @Manjula_Ganesap, The behaviour you're observing is likely due to the way Databricks handles schema changes while running a DLT pipeline. If your pipeline reads data with Auto Loader and the schema of the input data changes while an update is running, the update is logged as CANCELED and automatically retried. This behaviour was introduced in version 3.60 of Databricks, released between November 30 and December 6, 2021. In previous versions, when the input schema changed, the update would be logged as FAILED, even though Databricks automatically retries the update. This could be why your DLT initialization fails but succeeds upon automatic rerun.
Here is the relevant excerpt from the provided source:
"If your pipeline reads data with Auto Loader and the schema of the input data changes while an update is running, the update is logged as CANCELED and automatically retried. The new update is logged with the state SCHEMA_CHANGE. Previously, when the input schema changed, the update would be logged as FAILED even though automatically retries the update.
