How to identify or determine how many jobs will be performed if I submit code

[Sagar1]

I’m not able to find a source where it explains how to determine how many job a written piece of pyspark code will trigger. Can you please help me here. About stages I know that the number of shuffles equals to the number of stages.

[Debayan]

Hi @sagar Varma​ , Your job can consist of a single task or can be a large, multi-task workflow with complex dependencies.

You can list the jobs. Please refer: https://docs.databricks.com/workflows/jobs/jobs.html#view-jobs

[-werners-]

Hi @sagar Varma​ , I believe you are talking about Spark jobs (and not the Databricks workflow jobs).

Basically a spark job is created per physical operator. This can be an action but also others like BroadcastHashJoinExec.

Jacek Laskowski explained it on SO pretty well:

https://stackoverflow.com/questions/49385724/how-to-know-the-number-of-spark-jobs-and-stages-in-broa...

[Anonymous]

Hi @sagar Varma​ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!