Different result between manual and automated task run

[Harispap]

I have a notebook where I bring info about a previous task run metadata from the API ".... /jobs/runs/get". The response should be a dictionary that contains information such as task key, run if, run page URL etc. 

When I run the notebook as part of a scheduled ETL workflow the notebook failed with a key error that the 'result state' key was not found in the 'state' dictionary. However when I run it manually, the notebook didn't fail and the 'result state' key contained a valid value.

How is it possible to have different result between manual and automated task run?

Is there any case that the state of the job or tasks might have been different when the notebook was executed automatically compared to when I ran it manually. For example, the job might have completed and populated all the expected keys by the time I ran it manually.

[Kaniz]

Hi @Harispap, 

Yes, it’s possible to have different results between a manual and an automated task run. This could be due to a few reasons:

• The state of the job or tasks might have been different when the notebook was executed automatically compared to when you ran it manually. For example, the job might have completed and populated all the expected keys by the time you ran it manually.
• Databricks has different versions of their Jobs API¹. If your automated task is using a different version of the API than your manual task, this could result in different responses.
•  If you receive a 500-level error when making Jobs API requests, Databricks recommends retrying requ...¹. It’s possible that your manual run happened to avoid these errors while the automated task did not.
• Ensure you authenticate using a personal access token (PAT) and include it as the Bearer token in yo...².

To debug this issue, you could log the API response in both the manual and automated runs to see if there are any differences. Also, ensure that your automated task is using the same version of the API as your manual task. If the issue persists, you might want to reach out to us for further assistance.