topic Re: How to identify or determine how many jobs will be performed if I submit code in Data Engineering

How to identify or determine how many jobs will be performed if I submit code

Sagar1 — Sun, 16 Oct 2022 15:08:10 GMT

I’m not able to find a source where it explains how to determine how many job a written piece of pyspark code will trigger. Can you please help me here. About stages I know that the number of shuffles equals to the number of stages.

Re: How to identify or determine how many jobs will be performed if I submit code

-werners- — Wed, 19 Oct 2022 05:55:41 GMT

Hi @sagar Varma , I believe you are talking about Spark jobs (and not the Databricks workflow jobs).

Basically a spark job is created per physical operator. This can be an action but also others like BroadcastHashJoinExec.

Jacek Laskowski explained it on SO pretty well:

https://stackoverflow.com/questions/49385724/how-to-know-the-number-of-spark-jobs-and-stages-in-broadcast-join-query

Re: How to identify or determine how many jobs will be performed if I submit code

Anonymous — Sun, 27 Nov 2022 13:06:51 GMT

Hi @sagar Varma

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!

Re: How to identify or determine how many jobs will be performed if I submit code

Debayan — Tue, 18 Oct 2022 07:17:33 GMT

Hi @sagar Varma , Your job can consist of a single task or can be a large, multi-task workflow with complex dependencies.

You can list the jobs. Please refer: https://docs.databricks.com/workflows/jobs/jobs.html#view-jobs