topic Re: Intermittent failure with Python IMPORTS statements after upgrading to DBR18.0 in Data Engineering

Intermittent failure with Python IMPORTS statements after upgrading to DBR18.0

thackman — Thu, 12 Mar 2026 14:35:03 GMT

We have a python module (WidgetUtil.py) that sits in the same folder as our notebook. For the past few years we have been using a simple import statement to use it. Starting with DBR18.0 the imports fails intermittently (25% of the time) when running from job compute in PROD. It is 100% reliable when I use a personal/dedicated compute cluster in DEV. We rolled back to DBR17.2 and the failures went away. Then we rolled forward to DBR18.1Beta and the job started failing again. FYI: This is on Azure.

I did some debugging with AI suggestions, the theory was that FUSE was slow to mount. In the end that wasn't the case. We added a gatekeeper notebook at the start of the job. It monitored the paths and waited for the FUSE mount to complete. What we found was that the directory was always immediately available and we could either read the file immediately or it was never readable. Waiting up to two minutes never fixed the issue.

A job that succeeded.

A job that failed

Why is importing a .py file unreliable now?

Re: Intermittent failure with Python IMPORTS statements after upgrading to DBR18.0

Fabricio_Mattos — Thu, 12 Mar 2026 16:09:01 GMT

The issue is caused by changes in Databricks Runtime 18.x that make importing a plain.py file from the notebook’s folder unreliable on job compute, even though the same pattern still works consistently on a personal DEV cluster. In 18.x, the folder that contains your notebook (and WidgetUtil.py) is no longer consistently added to Python’s sys. path for jobs, so import WidgetUtils sometimes works and sometimes fails, even though the file is present and readable.

Explicitly add the module folder to sys. path

Use this when you want minimal structural changes and a fast fix.

import os import sys

module_dir = "/Workspace/Shared/prod_utils" i

f module_dir not in sys.path:

sys.path.insert(0, module_dir)

import WidgetUtil

Re: Intermittent failure with Python IMPORTS statements after upgrading to DBR18.0

thackman — Thu, 19 Mar 2026 18:36:51 GMT

Thanks for the suggestion Fabricio. We tried your suggestion of using sys.path.insert and it didn't improve the reliability. We found that converting some of the modules into notebooks improved reliability a lot. But other python modules we couldn't convert to notebooks because they were used in python udfs and we ran into pickle issues. Also, 18.1 is out of beta and it seemed slightly better than 18.0.

So overall our job now crashes 1-3 times per day with 80% of our python modules converted to notebooks and any remaining modules use imports and have a sys.path.insert block before them.

Re: Intermittent failure with Python IMPORTS statements after upgrading to DBR18.0

emma_s — Wed, 25 Mar 2026 17:57:50 GMT

Hi,

This is a known issue with the WSFS FUSE layer in DBR 18.x — a fix has been developed but may not
be fully rolled out yet. The most reliable workaround is to package your .py modules as a wheel
and install via %pip install, which bypasses FUSE entirely. If you need to stay on 18.x, raise a
support ticket referencing this behavior for engineering to check your region's patch status.

Thanks,

Emma