How to catch exception from dbutils.widgets.get(...)

entimaniac — Mon, 09 Aug 2021 16:16:59 GMT

I'm trying to write python notebook code that can be run from databricks web ui or from airflow. I intend to pass parameters from airflow via the job api using

notebook_params

. From what I understand, these are accessible as widget values.

dbutils.widgets.get

does not appear to accept a default value option so I assume I'll just have to catch an exception in case there isn't a value available. The code thrown on the UI (

InputWidgetNotDefined

) doesn't appear to work when I try catching it:

try:
  test_val = dbutils.widgets.get("test_name")
except InputWidgetNotDefined:
  test_val = "some default value"
print(test_val)

I can do a general, empty

except:

which will work but is generally bad practice to do blanket catches if we can avoid it.

try:
  test_val = dbutils.widgets.get("test_name")
except:
  test_val = "some default value"
print(test_val)

So is there a more precise exception I can catch?

Re: How to catch exception from dbutils.widgets.get(...)

blt — Tue, 24 May 2022 04:49:16 GMT

I handle this exception with something like:

import py4j
 
try:
  value = dbutils.widgets.get("parameter")
except py4j.protocol.Py4JJavaError as e:
  print(e)

If you look more closely at the stack trace you'll see the origin of the message is from some Java code:

/databricks/spark/python/lib/py4j-0.10.9.1-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
    325             if answer[1] == REFERENCE_TYPE:
--> 326                 raise Py4JJavaError(
    327                     "An error occurred while calling {0}{1}{2}.\n".
 
Py4JJavaError: An error occurred while calling o318.getArgument.
: com.databricks.dbutils_v1.InputWidgetNotDefined: No input widget named parameter is defined

"com.databricks.dbutils_v1.InputWidgetNotDefined" appears to be a Java class and the actual Python exception to handle is "Py4JJavaError".

topic Re: How to catch exception from dbutils.widgets.get(...) in Data Engineering

How to catch exception from dbutils.widgets.get(...)

Re: How to catch exception from dbutils.widgets.get(...)