topic RESOURCE_EXHAUSTED dbutils.jobs.taskValues.get in Data Engineering

RESOURCE_EXHAUSTED dbutils.jobs.taskValues.get

leireroman — Thu, 08 Aug 2024 07:53:15 GMT

I've a job in Databricks running multiple tasks in parallel. Those tasks read parameters of the job using the utility of dbutils. I'm getting the following error when trying to read parameters in my different tasks:

com.databricks.common.client.DatabricksServiceHttpClientException: RESOURCE_EXHAUSTED: Too many requests. Please wait a moment and try again. If the issue persists, consider adjusting your request frequency or reaching out to support for assistance.

This has been working well for a long time, but today it failed. I tried the job a second time, and it failed again. Any help on why it failed?

Re: RESOURCE_EXHAUSTED dbutils.jobs.taskValues.get

kesi — Thu, 08 Aug 2024 20:18:37 GMT

We have also had a RESOURCE_EXHAUSED error this morning in APAC on AWS. The job has 49 parallel tasks and has been running without an issue for maybe 6 months.

Re: RESOURCE_EXHAUSTED dbutils.jobs.taskValues.get

joachimswig — Fri, 09 Aug 2024 00:31:55 GMT

We've been seeing this as well, on a job with ~50 parallel tasks which had been running without issue for the past year

Re: RESOURCE_EXHAUSTED dbutils.jobs.taskValues.get

leireroman — Fri, 16 Aug 2024 05:36:49 GMT

Hi all,

Our solution has been to use job parameters and dynamic value references. These are read using dbutils.widgets.get() instead of dbutils.jobs.taskValues.get(). Now, our ETL is working well again.

Pass context about job runs into job tasks - Azure Databricks | Microsoft Learn

Configure settings for Azure Databricks jobs - Azure Databricks | Microsoft Learn

Hope you find this information useful.