topic Re: How to optimize jobs performance in Data Engineering

How to optimize jobs performance

RajeshRK — Fri, 17 Mar 2023 11:29:09 GMT

Hi Team,

We have a complex ETL job running in databricks for 6 hours. The cluster has the below configuration:

Minworkers: 16

Maxworkers: 24

Worker and Driver Node Type: Standard_DS14_v2. (16 cores, 128 GB RAM)

I have monitored the job progress in Spark UI for an hour, and my observations are below:

- The jobs are progressing and not stuck for a long time.

- The workers nodes scaled up to 24 (max_workers configured)

- Shuffling (Read/Write) happens with a large amount of data. (I Ran this job with spark.sql.shuffle.partitions 4000)

We are expecting the jobs should be completed within 4 hours. Any suggestions, please, to optimize the performance of the job?

Regards,

Rajesh.

Re: How to optimize jobs performance

Lakshay — Fri, 17 Mar 2023 14:35:33 GMT

Hi @Rajesh Kannan R , Can you check the Spark UI for the spark job where the job is spending most of the time. Also, look for any failed spark jobs in Spark UI.

Re: How to optimize jobs performance

RajeshRK — Fri, 17 Mar 2023 14:47:19 GMT

Hi Lakshay,

Thank you for replying. One thing I noticed is in the job description in "Spark UI", each job with the below code takes an average of 15 minutes.

"save at StoreTransform.scala"

Not sure whether it is a custom code or a Databricks code.

Regards,

Rajesh.

Re: How to optimize jobs performance

Lakshay — Fri, 17 Mar 2023 14:55:14 GMT

Hi @Rajesh Kannan R , It looks like a custom code. Could you please share a task-level screenshot of one of these stages?

Re: How to optimize jobs performance

RajeshRK — Fri, 17 Mar 2023 15:03:25 GMT

Hi Lakshay,

Unfortunately, I haven't captured it. I will share if I run the job next time.

Regards,

Rajesh.

Re: How to optimize jobs performance

Lakshay — Fri, 17 Mar 2023 19:06:13 GMT

Sure. You can also try the below suggestions:

Use compute-optimized Node Type. Currently, you are using a Memory-optimized one.
Run the job with spark.sql.shuffle.partitions auto

Re: How to optimize jobs performance

Anonymous — Sat, 18 Mar 2023 06:08:46 GMT

Hi @Rajesh Kannan R

Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.

Please help us select the best solution by clicking on "Select As Best" if it does.

Your feedback will help us ensure that we are providing the best possible service to you.

Thank you!

Re: How to optimize jobs performance

RajeshRK — Tue, 21 Mar 2023 06:54:37 GMT

@Lakshay Goel

Hi Lakshay,

It takes a couple of days to test this recommendation. I will try the job execution with new recommendations and update this thread.

Regards,

Rajesh.