Hi Anil,

How are you doing today?, As per my understanding, When you move Spark jobs from a self-hosted YARN cluster to Databricks and run them using Spark JARs on job compute clusters, it's normal to see a few extra stages added in the job execution plan. These stages are usually due to Databricks’ built-in features like adaptive query execution (AQE), automatic optimizations, or internal tracking. While these help in performance tuning, they can sometimes increase the total runtime if not tuned well. I’d suggest trying to disable AQE temporarily (spark.sql.adaptive.enabled to false) and reviewing the job stages in the Spark UI to see what’s taking time. Also, double-check if broadcast joins or data skew might be causing shuffle delays. Using compute pools can also reduce cold-start delays if you're launching new clusters for each run. A bit of tuning here can make a big difference — happy to help further if you share a specific job plan or logs!

Regards,

Brahma