topic Re: databricks_error_message:REQUEST_LIMIT_EXCEEDED: in Data Engineering

databricks_error_message:REQUEST_LIMIT_EXCEEDED:

prashantjjain33 — Wed, 02 Apr 2025 07:22:33 GMT

A databricks job failed unexpectedly with below error. There ware only 5 jobs running at that time and no major operations. what could be the root cause and how can we avoid this in future

Cluster '0331-xxxxxx-zs8i8pcn' was terminated. Reason: INIT_SCRIPT_FAILURE (CLIENT_ERROR). Parameters: instance_id:i-097022eeexxxxx7, databricks_error_message:REQUEST_LIMIT_EXCEEDED: Your workspace(xxxxxxx10399) is sending too many requests..

Re: databricks_error_message:REQUEST_LIMIT_EXCEEDED:

saurabh18cs — Wed, 02 Apr 2025 08:17:31 GMT

are you generating any databricks tokens in this process? if yes there is a limit of 600.

Re: databricks_error_message:REQUEST_LIMIT_EXCEEDED:

prashantjjain33 — Wed, 02 Apr 2025 08:53:30 GMT

Do you mean databricks PAT? No, we are not generating DBX PAT in this workflow

Re: databricks_error_message:REQUEST_LIMIT_EXCEEDED:

saurabh18cs — Wed, 02 Apr 2025 11:18:47 GMT

but this error indicates that your Databricks workspace is hitting API rate limits. (when there are too many API calls being made in a short period of time).

are you using multiple init scripts? can you Combine multiple init scripts into a single script to reduce the number of API calls during cluster startup.