INVALID_STATE: Storage configuration limit exceeded, only 11 storage configurations are allowed

[ktsoi]

Our team are trying to set up a new workspace (8th workspace), but failed to create the storage configurations required for the new workspace with an error of INVALID_STATE: Storage configuration limit exceeded, only 11 storage configurations are allowed.

What we have done:

For every workspace, we need 2 storage configurations (one for the root bucket (https://docs.databricks.com/administration-guide/account-api/aws-storage.html), and one for the audit log bucket (https://docs.databricks.com/administration-guide/account-settings/audit-logs.html)).

On the 18th January, we managed to successfully set up our 8th workspace (along with 16 total storage configurations). However due to some internal issue we decided to take it down and set up our 8th workspace again on the 31st January. This time the above error showed.

We have tried to set up using the Account Console, Databricks Account API, and DatabricksTerraform Provider, and all failed with the same error above.

We suspect that the quota imposed by Databricks has been changed, however from what I found on Databricks' documentation (https://docs.databricks.com/resources/limits.html), the storage configuration limit should be 55 per account. Would someone have more knowledge on this matter please?

Thanks in advance.

[Anonymous]

@Ktogofst​ :

It seems like you have reached the storage configuration limit for your Databricks account. The limit for storage configurations per account is currently set to 55, as mentioned in the Databricks documentation you referenced.

However, it is possible that there might be other limits or restrictions in place that are not documented publicly. I would recommend reaching out to Databricks support for assistance with this issue.

In the meantime, you can try deleting any unused or unnecessary storage configurations to free up space for the new workspace. You can also consider consolidating storage configurations where possible, for example, by using the same bucket for multiple workspaces.

If you are still unable to create the new workspace, Databricks support should be able to provide more information on the specific limit or restriction that is causing the issue and help you resolve it.

[ktsoi · Accepted solution]

Thank you, we resolved the issue directly with Databricks. It seemed that the storage configuration limit was not tiered (i.e. 55 per account regardless of the account type) a few months ago but has been updated to be tiered as now seen in the documentation (3/11/55 depending on the account type) so the 55 limit is only available for enterprise customers.

We are also considering consolidating the audit log buckets to reduce the number of storage configurations needed.

[Anonymous]

Hi @Kamting Tsoi​ 

Hope everything is going great.

Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we can help you. 

Cheers!

[_Architect_]

I solved the issue by simply going into Cloud Resources in Databricks console and navigated to "Credential Configuration" and "Storage Configuration" and deleted all the configurations which are not needed anymore(belongining to deleted workspaces)

I was able to successfully create workspace after that