cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

why the job running time and command execution time not matching in databricks

Dineshkumar_Raj
New Contributor

I have a azure databricks job and it's triggered via ADF using a API call. I want see why the job has been taking n minutes to complete the tasks. When the job execution results, The job execution time says 15 mins and the individual cells/commands doesn't add up to even 4-5 mins

The interactive cluster is already up and running while this got triggered. Please tell me why this sum of individual cell execution time doesn't match with the overall job execution time ? Where can I see what has taken the additional time here ?

2 REPLIES 2

Prabakar
Esteemed Contributor III
Esteemed Contributor III

The command execution is the time taken for the spark to process the task. However, after execution, the platform has to gather all the processed data, collate it and then push it for display in the jobs run result. So there should be some time difference between the execution time and job duration. A job has to go through a sequence of steps to do the execution and complete the job

Simply say,

notebook cell/command execution time = the time taken by spark to process the data and perform the task.

The job execution time = the time taken by spark + additional steps for the platform to start+process+complete the job.

Anonymous
Not applicable

Hey there @DineshKumar​ 

Does @Prabakar Ammeappin​'s response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly? Else please let us know if you need more help. 

Cheers!

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!