I've a job in Databricks running multiple tasks in parallel. Those tasks read parameters of the job using the utility of dbutils. I'm getting the following error when trying to read parameters in my different tasks:
com.databricks.common.client.DatabricksServiceHttpClientException: RESOURCE_EXHAUSTED: Too many requests. Please wait a moment and try again. If the issue persists, consider adjusting your request frequency or reaching out to support for assistance.
This has been working well for a long time, but today it failed. I tried the job a second time, and it failed again. Any help on why it failed?