Debugging a workflow

[Sega2]

Hi All,

With the new availability of debugging notebooks are really great. But how do you debug a workflow? Any suggestions on recommended practices on how to debug workflows? 

Best R, Thomas

[MuthuLakshmi]

@Sega2 

Please refer to this doc and confirm if it's helpful
https://docs.databricks.com/en/jobs/repair-job-failures.html#re-run-failed-and-skipped-tasks

[Sega2]

I am not trying to fix a failed job, but trying to find an approach to debug a workflow line by line. You can debug a note book, but how do you debug a workflow? 

[szymon_dybczak]

Hi @Sega2 ,

You can't debug from workflows UI. If notebook/python code failed in workflow, you need to manually find that notebook/module and then you can run it manually (passing all required parameters, if any) and use debugger from there.

[Sega2]

Thanks. I think we are getting closer now. What is the recommended practice for passing parameters in the notebook? In the notebook you have something like 

key1=dbutils.widgets.get("Key1") getting variables from the workflow. How do you simulate that in the notebook?

[szymon_dybczak · Accepted solution]

You can define widgets in your notebook in the following way (see screenshot). When databricks workflow is running it will pass all required parameters and you can use code you've provided to read these values key1=dbutils.widgets.get("Key1").
But, let's say that something went wrong and you want to debug. Than just fill parameters of widget with values and
and now you can debug without changing any code 😉

dbutils.widgets.text("param_source_schema", defaultValue="")
param_source_schema = dbutils.widgets.get("param_source_schema")

[Sega2]

That makes perfectly sense. Seems like thats the way to go. Thanks alot appreciated. I am new to Databrix how do you normally debug as an experienced DBX developer? Databricks connect or just in the browser and built in UI?