Hi @Upen_databricks, The issue you're experiencing with your DLT pipeline could be due to a couple of factors:
1. Development Optimizations: As per the Databricks release notes from September 7-13, 2021, new pipelines run in development mode by default. This mode optimizes pipeline execution by reusing clusters and turning off automatic retries. If your pipeline was created after this change and is running in development mode, reusing clusters could potentially cause longer provisioning times if the clusters are busy with other tasks.
2. Schema Changes: According to the release notes from November 30 - December 6, 2021, if your pipeline reads data with Auto Loader and the schema of the input data changes while an update is running, the update is logged as CANCELED and automatically retried. This could cause longer job times and even job failures if the schema changes are significant or frequent.
3. Bug Fixes: A bug caused the DLT UI to show a pipeline in a RUNNING state even after update failure. This bug has been fixed in a recent release. If your pipeline was affected by this bug, it could explain the job failures you're seeing.
You may want to consider these factors when troubleshooting your pipeline. If none of these seem to be the cause, you might want to contact Databricks support for further assistance.
