RESOURCE_EXHAUSTED dbutils.jobs.taskValues.get

[leireroman]

I've a job in Databricks running multiple tasks in parallel. Those tasks read parameters of the job using the utility of dbutils. I'm getting the following error when trying to read parameters in my different tasks:

com.databricks.common.client.DatabricksServiceHttpClientException: RESOURCE_EXHAUSTED: Too many requests. Please wait a moment and try again. If the issue persists, consider adjusting your request frequency or reaching out to support for assistance.

This has been working well for a long time, but today it failed. I tried the job a second time, and it failed again. Any help on why it failed?

[kesi]

We have also had a RESOURCE_EXHAUSED error this morning in APAC on AWS. The job has 49 parallel tasks and has been running without an issue for maybe 6 months.

[joachimswig]

We've been seeing this as well, on a job with ~50 parallel tasks which had been running without issue for the past year

[leireroman · Accepted solution]

Hi all,

Our solution has been to use job parameters and dynamic value references. These are read using dbutils.widgets.get() instead of dbutils.jobs.taskValues.get(). Now, our ETL is working well again.

Pass context about job runs into job tasks - Azure Databricks | Microsoft Learn

Configure settings for Azure Databricks jobs - Azure Databricks | Microsoft Learn

Hope you find this information useful.