serverless workflow Compute became unresponsive. Compute is likely out of memory.

[lizou1]

I set up run 10 notebooks at same time in serverless workflow

I got this error:

serverless workflow Compute became unresponsive. Compute is likely out of memory.

Is there a quota in serverless compute I can set in zure databricks? 

These notebooks are processing a few tables with 100 G in size each

[Advika · Accepted solution]

Hello @lizou1!

Yes, serverless compute has a quota, measured in Databricks Units (DBUs) per hour. For notebooks, jobs, or pipelines, there’s a built-in hourly spending upper limit.
To check your account’s serverless scale-up limits, you need to contact your Azure Databricks account team. You can also request a quota increase through the Azure Portal.

https://learn.microsoft.com/en-us/azure/databricks/admin/account-settings/serverless-quotas

[lizou1]

Sorry just FYI

 I am having this issue randomly every day, in environment 4 ,  I also have same issue in environment 2,3 as well

Run just waiting for hours

2031.048s info gc   GC(22) Pause Young (Allocation Failure) 1190M - 273M(3172M) 19.415ms

[lizou1]

The issue is new, and azure cloud providers are also not quite sure the details, will get more info later