This issue with %run in Databricks notebooks—where everything works interactively in the UI, but fails in a job context with java.util.NoSuchElementException: None.get—is a relatively common pain point for users leveraging notebook modularization. The error typically signals a problem with job execution context, notebook state initialization, or workspace scoping, rather than classic code or path issues.

Why This Happens

• Context Differences: When running manually, notebook execution is stateful and allows nested %run calls to inherit context (e.g., widgets, environment variables). Job runs, however, are stateless and reinitialize context per task, not reliably propagating state across nested %run invocations.

• Nested %run Limitations: Deeply nested %run statements, especially across multiple notebooks, can result in unpredictable behavior, as only top-level %run receives the initial widget/context setup. Lower-level notebooks may expect context or imports that aren't actually available in job execution.

• Widget/Parameter Initialization: While you report no missing widgets, jobs sometimes fail to initialize notebook widgets as they would in interactive mode, meaning that a called notebook may attempt to access a widget that’s not set.

Best Practices & Workarounds

• Avoid Deep Nesting of %run: Flatten your notebook structure as much as possible; ideally, have only the main notebook call all dependency notebooks directly (not via chains of %run).

• Use Python Modules for Shared Code: Instead of chaining functional notebooks with %run, convert common functionality into Python modules or scripts, and use standard import statements. This pattern is more robust and compatible with both interactive and job contexts.

• Check and Initialize Widgets Explicitly: In job contexts, always set widgets explicitly (using dbutils.widgets) rather than relying on inheritance or implicit initialization from interactive runs.

• Preview Job Cluster Configuration: Ensure that paths and notebook references are correct, and are accessible to the cluster executing the job (sometimes workspace paths work interactively, but fail in jobs due to cluster scope differences).

Troubleshooting Steps

• Reproduce the issue with reduced nesting—run the second notebook as a direct job task.

• Add explicit widget initialization checks and verbose logging in the included notebooks.

• Refactor shared code into reusable Python files stored in DBFS or workspace repo, then import in the main notebook.