Hi,
We have been planning to migrate the Synapse Data bricks activity executions from 'All-purpose cluster' to 'New job cluster' to reduce overall cost. We are using Standard_D3_v2 as cluster node type that has 4 CPU cores in total. The current quota for Standard Dv2 family is 50 vCPU.
As per the calculation, Number of worker nodes= Total vCPU quota/vCPUs per node.
In this case: 50 vCPUs / 4 vCPUs per node = 12.5
Since we can't have a fraction of a node, we have selected 12 worker nodes while creating Linked Service for new job cluster.
While running the synapse pipeline, we are getting this error- "Reason: INVALID_ARGUMENT (CLIENT_ERROR). Parameters: data bricks _error_ message: Operation could not be completed as it results in exceeding approved standardDv2Family Cores quota."
Attaching the screenshot of the error message for your reference:
It would be great if someone could check this issue and provide me with your suggestions.
Hi,
We are currently working on resolving the issue without exceeding the quota, if possible.
Here are our observations:
1. Notebook Execution Time Difference:
- When running a notebook activity using the interactive cluster, it completed in 5 minutes and 7 seconds. However, the same notebook took 1 hour and 32 minutes to execute using the job cluster.
Attaching the configuration details for both the clusters:
Interactive Cluster:
New Job Cluster:
2. Job Cluster having Node Type Standard_D8ds_v4 :
- We are getting an error ‘Node type Standard_D8ds_v4 is not supported’ when we tried using new job cluster having cluster node type as Standard_D8ds_v4, however it’s available in the list we are getting in the error message.
It would be helpful if you could check these issues and provide us with your suggestions.
