topic How to get the last time ( previous ) databricks job run time? in Data Engineering

How to get the last time ( previous ) databricks job run time?

sgannavaram — Mon, 07 Mar 2022 05:27:22 GMT

How to get the last databricks job run time? I have a requirement where i need to pass last job runtime as an argument in SQL and this SQL get the records from snowflake database based on this timestamp.

Re: How to get the last time ( previous ) databricks job run time?

Hubert-Dudek — Mon, 07 Mar 2022 10:39:01 GMT

You can get last job run time using Jobs API: https://docs.databricks.com/dev-tools/api/latest/jobs.html#operation/JobsRunsList

https://<databricks-instance>/api/2.1/jobs/runs/list

https://<databricks-instance>/api/2.1/jobs/runs/get

Re: How to get the last time ( previous ) databricks job run time?

sgannavaram — Mon, 07 Mar 2022 14:53:13 GMT

Thank you very much for your response Hubert. How can we call these api in databricks code? it would be really help if you have any sample code.

Re: How to get the last time ( previous ) databricks job run time?

RKNutalapati — Tue, 08 Mar 2022 03:21:02 GMT

@Srinivas Gannavaram : Sample python pseudocode for jobs API. Hope this helps you to build your Logic

import requests
import json
 
databricks_instance ="<databricks-instances>"
url_list = f"{databricks_instance}/api/2.1/jobs/runs/get?run_id=<Your Job RunID>"
 
headers = {
 'Authorization': 'Bearer <databricks-access-token>',
 'Content-Type': 'application/json'
}
 
response = requests.request("GET", url_list, headers=headers).json()
print(response)
print(response['job_id'])
print(response['start_time'])
print(response['end_time'])

Re: How to get the last time ( previous ) databricks job run time?

sgannavaram — Tue, 08 Mar 2022 04:05:15 GMT

Thank you so much. I will test it.

Re: How to get the last time ( previous ) databricks job run time?

Anonymous — Tue, 08 Mar 2022 21:20:44 GMT

@Srinivas Gannavaram - Please let us know how it goes!

Re: How to get the last time ( previous ) databricks job run time?

Anonymous — Fri, 22 Apr 2022 14:08:18 GMT

Hey there @Srinivas Gannavaram

Hope you are well. Just wanted to see if you were able to find an answer to your question and would you like to mark an answer as best? It would be really helpful for the other members.

Cheers!