dbutils.notebook.run() fails with job aborted but running the notebook individually works
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-03-2022 08:53 AM
I have a notebook that runs many notebooks in order, along the lines of:
```
%python
notebook_list = ['Notebook1', 'Notebook2']
for notebook in notebook_list:
print(f"Now on Notebook: {notebook}")
try:
dbutils.notebook.run(f'{notebook}', 3600)
except Exception as e:
print(e)
pass
```
When I run this notebook, I get job aborted exceptions amounting to the following:
```
Exceptions:
(1): Task failed while writing rows,
(2): Failed to execute user defined function (SQLDriverLocal$$$Lambda$1707/462935920: (string) => string)
(3): No input widget named effective_dating is defined
```
For #2 -- I have no user defined functions; I only use Spark functions.
However, if I run `Notebook1` or `Notebook2` (Run all button) individually, they run as expected.
To make matters more intriguing, if I run the loop above for the *second* time, each notebook will run fine; but **only after** I run each individually.
I would like to have a single notebook that runs many notebooks, rather than setup a pipeline to run each individual notebook on its own.
Running the notebooks as a Workflow/Job also returns the same issue.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-04-2022 02:18 AM
hi @Andrew Fogarty could you please share the stacktrace? We need to test this to understand what went wrong. If you have a minimal repro, could you please export the .dbc file and add it here?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-04-2022 04:07 AM
Yes! FailDemo is the .dbc of the notebook loop. RIght after this failure at 7:00:25 AM, I ran the notebook individually with no errors.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-04-2022 05:12 AM
I found the problem. Even if a notebook creates and specifies a widget fully, the notebook run process, e.g, dbutils.notebook.run('notebook') will not know how to use it. If I replace my widget with a non-widget provided value, the process works fine.
How can I use my widgets without this failure?

