topic Re: ETL job execution failure in serverless compute in Data Engineering

ETL job execution failure in serverless compute

furqanraza — Tue, 03 Dec 2024 08:40:25 GMT

Hi,

We facing an issue when executing the ELT pipeline on serverless compute. In the ETL pipeline, for some users, a task gets stuck every time we run the job. However, a similar ETL pipeline works fine for other users. Furthermore, there are canceled queries in the query history, but it shows that these are still being processed.

Have you faced a similar issue in Databricks with serverless compute?

Re: ETL job execution failure in serverless compute

ozaaditya — Wed, 04 Dec 2024 05:59:19 GMT

Hi,

Could you please share the error message or any logs you’re seeing when the task gets stuck? This will help in diagnosing the issue and identifying potential solutions.

Re: ETL job execution failure in serverless compute

furqanraza — Wed, 04 Dec 2024 08:37:57 GMT

Hi,

The following are the screenshots of query history and query profile.