Intermittent error databricks job kept running

[abelian-grape]

Hi i have the following error, but the job kept running, is that normal?

{     "message": "The service at /api/2.0/jobs/runs/get?run_id=899157004942769 is temporarily unavailable. Please try again later. [TraceId: -]",     "error_code": "TEMPORARILY_UNAVAILABLE" }

[Ayushi_Suthar]

Hi @abelian-grape , 

Understood your concern, but could you please confirm the below details for more insights on this?

• From when you are observing this issue and how frequent is that?
• Have you observed this issue today, if yes kindly share the timestamp.
• Confirm if you have any firewall between the control plane and the data plane.

Also, this could be due to the Planned Maintenance and this issue would've been resolved while running the jobs post maintenance, for more details on this maintenance you could refer to the below Status Page. Please let us know if you find the details helpful. 

Status Page:
https://status.databricks.com/pages/maintenance/5cf02dde58a00904bda41926/6552782c3abe2404d1f5b284

Followups are appreciated.
Kudos
Ayushi

[abelian-grape]

@Ayushi_Suthar 

There have been 17 occurences since January 18th

The last time this happened was on 

2024-02-09 14:37:16

[abelian-grape]

@Ayushi_Suthar no fire wall between control plane and data plane

[abelian-grape]

@Ayushi_Suthar also when ever it happens the job status does not change to "failed". But it keeps running. Is that normal?

[Kaniz]

Hi @abelian-grape, It may just be a temporary setback that you can overlook for now. However, if your job seems to be stuck or making no progress, it's best to pause it and come back to it at a later time. 

Nervous about that strange [TraceId: -] in the error message? Don't worry, it just means that there isn't a specific trace ID linked to the error. But if you have access to additional logs or monitoring tools, it's worth taking a closer look to see if you can gather more information about this trace ID. Don't forget to also keep an eye out for any other error messages or warnings that may give you a better understanding of the situation.