Hey folks,
a quick check -
wanted to gather thoughts on how you manage demand for azure VM quota so you don't run into quota limits issues.
In our case, we have several data domains (finance, master data, supply chain...) executing their projects in Databricks. Most of them use, say, EDSv4 family series of machine for data processing via databricks jobs (non-serverless). We have requested say for 10000 vCPUs of quota from Azure. At times, we run out of quota even though we have alert on 75% of usage. One day, the quota reaches 100% limit.
We prepared dashboards using system tables to see which data domain is consuming how many vCPUs and which actually caused the quota limit; but, this is all reactive. We also like to put restrictions on VM types using compute policies.
Asking data domains to provide information around the demand (no of jobs going live as part of a new project in a month/quarter etc..) is not going to work in our environment, don't ask me why 😂.
Curious to see how others handle the cases...
