Intermittent failure with Python IMPORTS statements after upgrading to DBR18.0

[thackman]

We have a python module (WidgetUtil.py) that sits in the same folder as our notebook. For the past few years we have been using a simple import statement to use it. Starting with DBR18.0 the imports fails intermittently (25% of the time) when running from job compute in PROD. It is 100% reliable when I use a personal/dedicated compute cluster in DEV. We rolled back to DBR17.2 and the failures went away. Then we rolled forward to DBR18.1Beta and the job started failing again.  FYI: This is on Azure.

I did some debugging with AI suggestions, the theory was that FUSE was slow to mount. In the end that wasn't the case. We added a gatekeeper notebook at the start of the job. It monitored the paths and waited for the FUSE mount to complete. What we found was that the directory was always immediately available and we could either read the file immediately or it was never readable. Waiting up to two minutes never fixed the issue.

A job that succeeded.

A job that failed

Why is importing a .py file unreliable now? 

 

 

[Fabricio_Mattos]

The issue is caused by changes in Databricks Runtime 18.x that make importing a plain.py file from the notebook’s folder unreliable on job compute, even though the same pattern still works consistently on a personal DEV cluster. In 18.x, the folder that contains your notebook (and WidgetUtil.py) is no longer consistently added to Python’s sys. path for jobs, so import WidgetUtils sometimes works and sometimes fails, even though the file is present and readable.

Explicitly add the module folder to sys. path

Use this when you want minimal structural changes and a fast fix.

import os import sys

module_dir = "/Workspace/Shared/prod_utils" i

f module_dir not in sys.path:

   sys.path.insert(0, module_dir)

import WidgetUtil

[thackman]

Thanks for the suggestion Fabricio.  We tried your suggestion of using sys.path.insert and it didn't improve the reliability.  We found that converting some of the modules into notebooks improved reliability a lot. But other python modules we couldn't convert to notebooks because they were used in python udfs and we ran into pickle issues. Also, 18.1 is out of beta and it seemed slightly better than 18.0. 

So overall our job now crashes 1-3 times per day with 80% of our python modules converted to notebooks and any remaining modules use imports and have a sys.path.insert block before them. 

[emma_s · Accepted solution]

Hi,

This is a known issue with the WSFS FUSE layer in DBR 18.x — a fix has been developed but may not
be fully rolled out yet. The most reliable workaround is to package your .py modules as a wheel
and install via %pip install, which bypasses FUSE entirely. If you need to stay on 18.x, raise a
support ticket referencing this behavior for engineering to check your region's patch status.

Thanks,

Emma

[yoavmagor]

Hi, I'm having a similar (not identical) issue with FUSE and DBR17.3.

My Python imports fail when I run the code from a DBR17.3 cluster, but if I go back to 15 or 16 the imports succeed - and this applies to imports that I run in a Gir folder under my user folder. If i run the exact same code on DBR17.3 in a Repo Git folder (for the same Git repository), the import succeeds.

I can see the folder that I try to import from, but it appears that FUSE is mounting no files from it at all - resulting in the import failure.

[freddyT]

Hello, we also have the same issue. The error is sporadic (on the next executions it passes, then it fails again). It is either "cannot import name ..." or "no module ...".
We did not change anything and we have this issue since a few hours ago. We are on a job compute (Single node: Standard_DC4as_v5 · Release: 17.3.14).

[pjvi]

Hi, are you aware if this is also affecting Databricks Free Edition environment v5 ? I think I am having the same issue.

Thanks.