Widget management in Jupyter (ipynb) notebooks

[mkolonay]

Hello!

We use widgets to set values for which catalogs and schemas to use in our solutions. These values change as we change environments, feature, dev, uat, prod.

In dev, uat and prod, the values are set in workflows so the widget values are not an issue. We work on more than one feature branch for the same solution at a time and we use naming convention to avoid our developers stepping on each other's work. So, one feature branch will use feature_XXX schema and another will use feature_YYY schema. These values are set by a script that creates widgets values based on the branch they have checked out. 

I'd like to start using the Jupyter notebook format, but I see what widget values are included in the notebook. This will muddy up our commits, so I'd like to leave widgets, and their values out of source control. Are there any options to do this?

Thanks!

[Alberto_Umana]

Hi @mkolonay,

I can suggest two approaches:

1. Store widget values outside the notebook in a separate configuration file. When a notebook is executed, a script can dynamically read the configuration file and set the widget values accordingly. This approach ensures that the widget values are not embedded within the notebook file itself and therefore remain out of source control

2. Utilize dbutils.widgets and their fallback values:

* Local Implementation: Implement a mechanism to set default values from a local script or environment variables.
* Interactive Widgets: If the interactive environment (like Jupyter) does not set the widget values, the widgets should fall back to default values specified in the local script

[mkolonay]

Thanks for the quick response.

Your suggestion is essentially what we are currently doing. We have a config file for each environment that is used to create all the widgets. In the first cell of our notebooks, we call a script to create the widgets using the correct config file. Again, we only do this in our feature environments where our developers are working manually through the notebooks, the higher environments values are set in the workflow.

There is more going on around it but when you boil it down, we are doing this:

for key, value in config.items():
     dbutils.widgets.text(key,value)

Once I run the dbutils.widgets.text() I see changes in source control, check the attachment for an example.

Thanks!