topic Debugging a workflow in Data Engineering

Debugging a workflow

Sega2 — Mon, 18 Nov 2024 10:33:55 GMT

Hi All,

With the new availability of debugging notebooks are really great. But how do you debug a workflow? Any suggestions on recommended practices on how to debug workflows?

Best R, Thomas

Re: Debugging a workflow

MuthuLakshmi — Mon, 18 Nov 2024 11:18:25 GMT

@Sega2

Please refer to this doc and confirm if it's helpful
https://docs.databricks.com/en/jobs/repair-job-failures.html#re-run-failed-and-skipped-tasks

Re: Debugging a workflow

Sega2 — Mon, 18 Nov 2024 13:09:59 GMT

I am not trying to fix a failed job, but trying to find an approach to debug a workflow line by line. You can debug a note book, but how do you debug a workflow?

Re: Debugging a workflow

szymon_dybczak — Mon, 18 Nov 2024 13:15:02 GMT

Hi @Sega2 ,

You can't debug from workflows UI. If notebook/python code failed in workflow, you need to manually find that notebook/module and then you can run it manually (passing all required parameters, if any) and use debugger from there.

Re: Debugging a workflow

Sega2 — Mon, 18 Nov 2024 13:36:52 GMT

Thanks. I think we are getting closer now. What is the recommended practice for passing parameters in the notebook? In the notebook you have something like

key1=dbutils.widgets.get("Key1") getting variables from the workflow. How do you simulate that in the notebook?

Re: Debugging a workflow

szymon_dybczak — Mon, 18 Nov 2024 13:51:36 GMT

You can define widgets in your notebook in the following way (see screenshot). When databricks workflow is running it will pass all required parameters and you can use code you've provided to read these values key1=dbutils.widgets.get("Key1").
But, let's say that something went wrong and you want to debug. Than just fill parameters of widget with values and
and now you can debug without changing any code 😉

dbutils.widgets.text("param_source_schema", defaultValue="") param_source_schema = dbutils.widgets.get("param_source_schema")

Re: Debugging a workflow

Sega2 — Tue, 19 Nov 2024 16:24:04 GMT

That makes perfectly sense. Seems like thats the way to go. Thanks alot appreciated. I am new to Databrix how do you normally debug as an experienced DBX developer? Databricks connect or just in the browser and built in UI?