Hi Team,

We have a complex ETL job running in databricks for 6 hours. The cluster has the below configuration:

Minworkers: 16

Maxworkers: 24

Worker and Driver Node Type: Standard_DS14_v2. (16 cores, 128 GB RAM)

I have monitored the job progress in Spark UI for an hour, and my observations are below:

- The jobs are progressing and not stuck for a long time.  

- The workers nodes scaled up to 24 (max_workers configured)

- Shuffling (Read/Write) happens with a large amount of data.  (I Ran this job with spark.sql.shuffle.partitions 4000)

We are expecting the jobs should be completed within 4 hours. Any suggestions, please, to optimize the performance of the job?

Regards,

Rajesh.