cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Is there a way to list the dlt maintenance jobs through the API?

gustavomcarmo-h
New Contributor III

After creating the delta pipeline, I would like to get details from the dlt maintenance job automatically created by Databricks, like the scheduled time when the dlt maintenance tasks will be executed. However, it seems the Job API 2.1 doesn't cover this type of job in the list operation. Is there a workaround?

1 ACCEPTED SOLUTION

Accepted Solutions

gustavomcarmo-h
New Contributor III

Hi @Debayan Mukherjee​,

Although undocumented, I figured out that the Databricks Jobs API list endpoint has a `job_type` parameter that I can set `MAINTENANCE_PIPELINE` to get the DLTs maintenance jobs. The following command works: `curl -H "Authorization: Bearer $DATABRICKS_AAD_TOKEN" "https://$databricks_host/api/2.1/jobs/list?limit=25&job_type=MAINTENANCE_PIPELINE"`

Please update the Databricks Jobs API documentation!!!

View solution in original post

5 REPLIES 5

Debayan
Esteemed Contributor III
Esteemed Contributor III

Hi,

You can list DLT pipelines. https://learn.microsoft.com/en-us/azure/databricks/delta-live-tables/api-guide#--list-pipeline-event...

Please let us know if this helps. Also, please tag @Debayan Mukherjee​ with your next comment so I will be getting notified.

gustavomcarmo-h
New Contributor III

Hi @Debayan Mukherjee​,

Unfortunately this doesn't help. The Pipelines API events endpoint only covers update events, not maintenance events. I need details about DLTs maintenance tasks performed automatically by Databricks under the hood. The Jobs API list endpoint should also cover DLTs maintenance jobs created by Databricks automatically when delta pipelines are created. There must be a way for knowing which maintenance jobs exist.

gustavomcarmo-h
New Contributor III

Hi @Debayan Mukherjee​,

Although undocumented, I figured out that the Databricks Jobs API list endpoint has a `job_type` parameter that I can set `MAINTENANCE_PIPELINE` to get the DLTs maintenance jobs. The following command works: `curl -H "Authorization: Bearer $DATABRICKS_AAD_TOKEN" "https://$databricks_host/api/2.1/jobs/list?limit=25&job_type=MAINTENANCE_PIPELINE"`

Please update the Databricks Jobs API documentation!!!

Debayan
Esteemed Contributor III
Esteemed Contributor III

Thanks for the confirmation, we are glad that it has been fixed.

gustavomcarmo-h
New Contributor III

Hi @Debayan Mukherjee​ ,

Actually the Databricks Jobs API documentation has not been fixed yet. The parameter `job_type` should be included in the list endpoint request documentation. Please do this in order to avoid unnecessary questions here in the Databricks Community portal.

Thanks!