ETL job execution failure in serverless compute

[furqanraza]

Hi,

We facing an issue when executing the ELT pipeline on serverless compute. In the ETL pipeline, for some users, a task gets stuck every time we run the job. However, a similar ETL pipeline works fine for other users. Furthermore, there are canceled queries in the query history, but it shows that these are still being processed.

Have you faced a similar issue in Databricks with serverless compute?

[ozaaditya]

Hi,

Could you please share the error message or any logs you’re seeing when the task gets stuck? This will help in diagnosing the issue and identifying potential solutions.

[furqanraza]

Hi,

The following are the screenshots of query history and query profile.