Migration of Synapse Data bricks activity executions from All purpose cluster to New job cluster

[IshaBudhiraja]

Hi,

We have been planning to migrate the Synapse Data bricks activity executions from 'All-purpose cluster' to 'New job cluster' to reduce overall cost. We are using Standard_D3_v2 as cluster node type that has 4 CPU cores in total. The current quota for Standard Dv2 family is 50 vCPU.

As per the calculation, Number of worker nodes= Total vCPU quota/vCPUs per node.

In this case: 50 vCPUs / 4 vCPUs per node = 12.5

Since we can't have a fraction of a node, we have selected 12 worker nodes while creating Linked Service for new job cluster.

While running the synapse pipeline, we are getting this error- "Reason: INVALID_ARGUMENT (CLIENT_ERROR). Parameters: data bricks _error_ message: Operation could not be completed as it results in exceeding approved standardDv2Family Cores quota."

Attaching the screenshot of the error message for your reference:

It would be great if someone could check this issue and provide me with your suggestions.

[Kaniz]

Hi @IshaBudhiraja, 

• Quotas are used for different resource groups, subscriptions, accounts, and scopes. The number of cores for a particular region may be restricted by your subscription.
• To verify your subscription’s usage and quotas, follow these steps:
  • Go to the Azure Portal.
  • Select your subscription.
  • Navigate to Settings > Usage + Quotas.
  • Filter by Microsoft.Compute to find the total regional vCPU consumption.
  • If the consumption is at capacity, you’ll need to request an increase in the region’s core limit.
  • Use the New Quota Request Option to raise the quota limit.
• Submit a request for a quota increase. You can apply for an increase in your cores quota using this link.
• Ensure that you request an increase to at least 12 cores to meet your current requirements for the S.... 
  •  

[IshaBudhiraja]

Hi, Thanks for sharing these details.

Could you please suggest some other method to resolve this issue without exceeding the quota? Can we try it by some other method like adjusting number of worker nodes, batch count, etc.?

[IshaBudhiraja]

Hi,

We are currently working on resolving the issue without exceeding the quota, if possible.

Here are our observations:

1. Notebook Execution Time Difference:

• When running a notebook activity using the interactive cluster, it completed in 5 minutes and 7 seconds. However, the same notebook took 1 hour and 32 minutes to execute using the job cluster.

         Attaching the configuration details for both the clusters:

Interactive Cluster:

New Job Cluster:     

2. Job Cluster having Node Type Standard_D8ds_v4 :

• We are getting an error ‘Node type Standard_D8ds_v4 is not supported’ when we tried using new job cluster having cluster node type as Standard_D8ds_v4, however it’s available in the list we are getting in the error message.

It would be helpful if you could check these issues and provide us with your suggestions.