topic Re: Error Handling and Custom Messages in Workflows in Data Engineering

Error Handling and Custom Messages in Workflows

CaptainJack — Tue, 30 Jul 2024 14:44:03 GMT

I would like to be able to get custom error's message ideally visible from Workflows > Jobs UI.

1. For example, workflow failed because file was missing and could not find it, in this case I am getting "Status" Failed and "Error Code" RunExecutionError. If I hove cursor on Status's red icon, I am able to see "[RunExecutionError] Workload failed, see run output for details". Is there anyway to have custom error there?

2. I would like to use /api/2.1/jobs/runs/get to get some information about error. I did API call and check ['tasks'] key, but all somehow useful which I can see is 'result_state': TERMINATED, state_message = 'Workload failed, see run output for details'. Is it possible to have some custom parameter there which could indicate what error happen? I thought about tag but is it possible to create tag or any other variable while runing notebook task and then read it from runs/get API call?

Re: Error Handling and Custom Messages in Workflows

Walter_C — Tue, 30 Jul 2024 18:04:14 GMT

Custom error messages in the Workflows > Jobs UI or via the /api/2.1/jobs/runs/get API call are not directly supported. The error messages you're seeing, such as "RunExecutionError", are standard error codes that Databricks uses to indicate the type of error that occurred.

Re: Error Handling and Custom Messages in Workflows

CaptainJack — Wed, 31 Jul 2024 07:16:54 GMT

Any idea for workaround? I thought about tagValue / jobValue as error but I am not sure if it is possible to get these using jobs/runs/get API. If it is really not possible, I am thinking about email notifications as last resort.

Re: Error Handling and Custom Messages in Workflows

Edthehead — Fri, 02 Aug 2024 05:28:59 GMT

What you can do is pass the custom error message you want from the notebook back to the workflow

output = f"There was an error with {error_code} : {error_msg}"

dbutils.notebook.exit(output)

Then when you are fetching the status of your pipeline, you can also check the output of each task. You need to loop through each task runid to get the individual outputs. You would use jobs/runs/get-output with run_id as a parameter. The run_id should be that of the task and not the job itself. Based on the output of the task, your pipeline can understand the custom error. You will find the output in a json variable called

notebook_output.