topic Migration of Synapse Data bricks activity executions from All purpose cluster to New job cluster in Data Engineering

Migration of Synapse Data bricks activity executions from All purpose cluster to New job cluster

IshaBudhiraja — Fri, 29 Mar 2024 05:15:06 GMT

Hi,

We have been planning to migrate the Synapse Data bricks activity executions from 'All-purpose cluster' to 'New job cluster' to reduce overall cost. We are using Standard_D3_v2 as cluster node type that has 4 CPU cores in total. The current quota for Standard Dv2 family is 50 vCPU.

As per the calculation, Number of worker nodes= Total vCPU quota/vCPUs per node.

In this case: 50 vCPUs / 4 vCPUs per node = 12.5

Since we can't have a fraction of a node, we have selected 12 worker nodes while creating Linked Service for new job cluster.

While running the synapse pipeline, we are getting this error- "Reason: INVALID_ARGUMENT (CLIENT_ERROR). Parameters: data bricks _error_ message: Operation could not be completed as it results in exceeding approved standardDv2Family Cores quota."

Attaching the screenshot of the error message for your reference:

It would be great if someone could check this issue and provide me with your suggestions.

Re: Migration of Synapse Data bricks activity executions from All purpose cluster to New job cluster

IshaBudhiraja — Sun, 31 Mar 2024 07:33:07 GMT

Hi, Thanks for sharing these details.

Could you please suggest some other method to resolve this issue without exceeding the quota? Can we try it by some other method like adjusting number of worker nodes, batch count, etc.?

Re: Migration of Synapse Data bricks activity executions from All purpose cluster to New job cluster

IshaBudhiraja — Tue, 02 Apr 2024 19:15:08 GMT

Hi,

We are currently working on resolving the issue without exceeding the quota, if possible.

Here are our observations:

1. Notebook Execution Time Difference:

When running a notebook activity using the interactive cluster, it completed in 5 minutes and 7 seconds. However, the same notebook took 1 hour and 32 minutes to execute using the job cluster.

Attaching the configuration details for both the clusters:

Interactive Cluster:

New Job Cluster:

2. Job Cluster having Node Type Standard_D8ds_v4 :

We are getting an error ‘Node type Standard_D8ds_v4 is not supported’ when we tried using new job cluster having cluster node type as Standard_D8ds_v4, however it’s available in the list we are getting in the error message.

It would be helpful if you could check these issues and provide us with your suggestions.

Re: Migration of Synapse Data bricks activity executions from All purpose cluster to New job cluster

NandiniN — Thu, 09 Jan 2025 08:05:32 GMT

You can choose instance types based on the workload:

Memory Optimized instances for machine learning workloads, tasks involving a lot of shuffle and spills, and when Spark caching is needed.
Compute Optimized instances for structured streaming jobs, ELT with full scan and no data reuse, and to run OPTIMIZE and Z-order Delta commands.
Storage Optimized instances to leverage Delta caching, for ML and DL workloads with data caching, and for ad hoc and interactive data analysis.
General Purpose instances for jobs without specific requirements and to run VACUUM Delta command.

Re: Migration of Synapse Data bricks activity executions from All purpose cluster to New job cluster

NandiniN — Thu, 09 Jan 2025 08:06:10 GMT

I also see a difference in Photon, Enable Photon for workloads with large data scans, joins, aggregations, and decimal computations. Photon provides significant performance benefits over the standard Databricks Runtime.