%run notebook fails in Job mode with Py4JJavaError (None.get), but works in interactive notebook

[Akshay_Petkar]

Hi everyone,

I’m facing an issue when executing a Databricks job where my notebook uses %run to include other notebooks. I have a final notebook added as a task in a job, and inside that notebook I use %run to call another notebook that contains all my functions. That second notebook again calls another notebook using %run.

When I run the notebook manually in the UI, everything works perfectly. But when the same notebook runs as a job, it fails with a Py4JJavaError showing “java.util.NoSuchElementException: None.get”.

There are no missing widgets or paths because the exact same setup runs fine interactively.

Has anyone else seen this behavior where %run works fine manually but fails with this error in a job? Any suggestions or best practices would be really helpful.

Thanks!

[K_Anudeep]

Hello @Akshay_Petkar ,

• %run can break in Jobs due to path resolution—use relative paths (e.g., %run ./lib/utils) and keep all targets in the same Repo.

• Don’t rely on widget state. Define widgets only in the entry notebook and pass values explicitly (or switch to dbutils.notebook.run() with arguments={...} for clean, isolated calls).

• Avoid nested/chained%run and side-effect cells; %run literally inlines code and order matters.

• The recommended approach for a long term mitgatiation is to move shared code into a package in the Repo and import it instead of %run.

[mark_ott]

This issue with %run in Databricks notebooks—where everything works interactively in the UI, but fails in a job context with java.util.NoSuchElementException: None.get—is a relatively common pain point for users leveraging notebook modularization. The error typically signals a problem with job execution context, notebook state initialization, or workspace scoping, rather than classic code or path issues.

Why This Happens

• Context Differences: When running manually, notebook execution is stateful and allows nested %run calls to inherit context (e.g., widgets, environment variables). Job runs, however, are stateless and reinitialize context per task, not reliably propagating state across nested %run invocations.

• Nested %run Limitations: Deeply nested %run statements, especially across multiple notebooks, can result in unpredictable behavior, as only top-level %run receives the initial widget/context setup. Lower-level notebooks may expect context or imports that aren't actually available in job execution.

• Widget/Parameter Initialization: While you report no missing widgets, jobs sometimes fail to initialize notebook widgets as they would in interactive mode, meaning that a called notebook may attempt to access a widget that’s not set.

Best Practices & Workarounds

• Avoid Deep Nesting of %run: Flatten your notebook structure as much as possible; ideally, have only the main notebook call all dependency notebooks directly (not via chains of %run).

• Use Python Modules for Shared Code: Instead of chaining functional notebooks with %run, convert common functionality into Python modules or scripts, and use standard import statements. This pattern is more robust and compatible with both interactive and job contexts.

• Check and Initialize Widgets Explicitly: In job contexts, always set widgets explicitly (using dbutils.widgets) rather than relying on inheritance or implicit initialization from interactive runs.

• Preview Job Cluster Configuration: Ensure that paths and notebook references are correct, and are accessible to the cluster executing the job (sometimes workspace paths work interactively, but fail in jobs due to cluster scope differences).

Troubleshooting Steps

• Reproduce the issue with reduced nesting—run the second notebook as a direct job task.

• Add explicit widget initialization checks and verbose logging in the included notebooks.

• Refactor shared code into reusable Python files stored in DBFS or workspace repo, then import in the main notebook.