topic Re: dbutils.notebook.run() fails with job aborted but running the notebook individually works in Data Engineering

dbutils.notebook.run() fails with job aborted but running the notebook individually works

auser85 — Fri, 03 Jun 2022 15:53:23 GMT

I have a notebook that runs many notebooks in order, along the lines of:

```

%python

notebook_list = ['Notebook1', 'Notebook2']

for notebook in notebook_list:

print(f"Now on Notebook: {notebook}")

try:

dbutils.notebook.run(f'{notebook}', 3600)

except Exception as e:

print(e)

pass

```

When I run this notebook, I get job aborted exceptions amounting to the following:

```

Exceptions:

(1): Task failed while writing rows,

(2): Failed to execute user defined function (SQLDriverLocal$$$Lambda$1707/462935920: (string) => string)

(3): No input widget named effective_dating is defined

```

For #2 -- I have no user defined functions; I only use Spark functions.

However, if I run `Notebook1` or `Notebook2` (Run all button) individually, they run as expected.

To make matters more intriguing, if I run the loop above for the *second* time, each notebook will run fine; but **only after** I run each individually.

I would like to have a single notebook that runs many notebooks, rather than setup a pipeline to run each individual notebook on its own.

Running the notebooks as a Workflow/Job also returns the same issue.

Re: dbutils.notebook.run() fails with job aborted but running the notebook individually works

Prabakar — Sat, 04 Jun 2022 09:18:00 GMT

hi @Andrew Fogarty could you please share the stacktrace? We need to test this to understand what went wrong. If you have a minimal repro, could you please export the .dbc file and add it here?

Re: dbutils.notebook.run() fails with job aborted but running the notebook individually works

auser85 — Sat, 04 Jun 2022 11:07:19 GMT

Yes! FailDemo is the .dbc of the notebook loop. RIght after this failure at 7:00:25 AM, I ran the notebook individually with no errors.

Re: dbutils.notebook.run() fails with job aborted but running the notebook individually works

auser85 — Sat, 04 Jun 2022 12:12:07 GMT

I found the problem. Even if a notebook creates and specifies a widget fully, the notebook run process, e.g, dbutils.notebook.run('notebook') will not know how to use it. If I replace my widget with a non-widget provided value, the process works fine.

How can I use my widgets without this failure?