Rahiman
Databricks Partner

We had similar error for one the DLT pipeline, This could be some times because of compute size, we had increased compute size of server in your DLT pipelines, still we were seeing this error while processing very large file. 

we then added below parameter to the DLT pipeline configuration, as the default timeout is 120s which increased to 3600s, then the pipeline ran successfully

pipeline.timeout=3600s
pipeline.clusterShutdown.delay=120s