topic Re: Databricks DBR 18.1 access workspace files error in Data Engineering

Databricks DBR 18.1 access workspace files error

der — Tue, 17 Mar 2026 08:11:30 GMT

import json with open("/Workspace/Users/<USER>/config.json", "r") as f: config = json.load(f) print(config)

Throws following error

OSError: [Errno 5] Input/output error: '/Workspace/Users/<USER>/config.json'
[Trace ID: 00-874e2bc3d747c3611c0c4adf64f48620-ed365d30ae67cc7f-00]
---------------------------------------------------------------------------
OSError Traceback (most recent call last)
File <command-6408169302940046>, line 3
1 import json
----> 3 with open("/Workspace/Users/<USER>/config.json", "r") as f:
4 config = json.load(f)
6 print(config)

File /databricks/python/lib/python3.12/site-packages/IPython/core/interactiveshell.py:324, in _modified_open(file, *args, **kwargs)
317 if file in {0, 1, 2}:
318 raise ValueError(
319 f"IPython won't let you open fd={file} by default "
320 "as it is likely to crash IPython. If you know what you are doing, "
321 "you can use builtins' open."
322 )
--> 324 return io_open(file, *args, **kwargs)

OSError: [Errno 5] Input/output error: '/Workspace/Users/<USER>/config.json'

Did something changed with DBR 18.1 and workspace files access?

I see no limitation in documentation
https://docs.databricks.com/aws/en/files/#work-with-workspace-files

Re: Databricks DBR 18.1 access workspace files error

Ashwin_DSA — Tue, 17 Mar 2026 10:11:02 GMT

Hi @der ,

I have checked internally. No specific changes to workspace files access were introduced as part of DBR 18.1. The error seems to indicate that the mount/backend is unhealthy or blocked (network/NSG/cluster config), not that your Python code is wrong.

Can you try the below...

import os, getpass, subprocess, textwrap user = getpass.getuser() path = f"/Workspace/Users/{user}" # Does listing the directory itself fail? print("Listing:", path) print(subprocess.run(["ls", "-l", path], text=True, capture_output=True))

If "ls" itself throws the same error, the /Workspace mount is broken on this cluster.

You can also spin up a new cluster with minimal config (no custom VNet/NSG changes, no init scripts, no extra drivers) and run the same open("/Workspace/...") there. If it works on the clean cluster but not on the original one, it’s a cluster/network configuration issue.

A robust workaround is to store configs on DBFS instead of raw /Workspace/Users/.. However, if you must use the workspace files... the officially supported way is through the Workspace Files API / WorkspaceClient().files, rather than relying on FUSE paths. Use the SDK to download the file to local disk, then json.load it.

If the above doesn't resolve your issue, this is a good candidate for a Databricks support case. As you already have it, attach the trace ID, cluster ID, workspace URL, and confirm that any customer‑managed NSGs allow egress to workspace APIs (including /api/2.0/workspace-files/*).

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

Re: Databricks DBR 18.1 access workspace files error

der — Tue, 17 Mar 2026 10:54:08 GMT

Hi @Ashwin_DSA

I restarted the cluster and it worked again. If you do it multiple times, most times it works and sometimes it fails. 😟 We see this issue only with DBR 18.1 (Standard with Photon, Dedicated without Photon)

DBFS is deactivated.

Re: Databricks DBR 18.1 access workspace files error

Ashwin_DSA — Tue, 17 Mar 2026 13:46:42 GMT

Hi @der,

Can you run the same notebook on a test cluster in the same workspace with DBR 18.0 (or 17.3 LTS) with same access mode, DBFS still disabled? If the issue disappears there and only occurs on 18.1, that’s strong evidence for an 18.1‑specific issue.

If you hit the error on 18.1.. just capture the trace ID from the stack trace and immediately run the below in a Python cell.

import os, getpass, subprocess user = getpass.getuser() base = f"/Workspace/Users/{user}" print("ls base:") print(subprocess.run(["ls", "-l", base], text=True, capture_output=True)) print("ls file:") print(subprocess.run(["ls", "-l", "/Workspace/Users/<USER>/config.json"], text=True, capture_output=True))

In a %sh cell, check for FUSE / workspace-files errors:

dmesg | tail -n 50 || true

You can then open a support ticket with the Workspace URL and region, Cluster ID, DBR: 18.1, access mode, DBFS disabled, the full Python stack trace with trace ID(s), Output of the ls / dmesg checks above on a failing run and also provide confirmation that the same code on 18.0 or 17.3 LTS does not show Errno 5.

That gives support/engineering exactly what they need to confirm a regression in 18.1’s WSFS/FUSE behaviour.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

Re: Databricks DBR 18.1 access workspace files error

Youry — Tue, 17 Mar 2026 13:47:04 GMT

We have similar issue with accessing Workspace files exactly in DBR 18.1 ( previous DBR 17.xx have worked even with relative paths OK ).

import yaml import os def settings(): try: script_dir = os.path.dirname(os.path.abspath(__file__)) except NameError: script_dir = os.getcwd() config_path = os.path.normpath( os.path.join(script_dir, "..", "config", "config.yml") )

at the end OSError: [Errno 5] Input/output error: '/Workspace/code/config/config.yml' [Trace ID: 00-188a04ab49de4ed3b4ab6e0e3ba17e58-b94fcf438d525fb0-00]
Looks file cannot be opened even it's FDQN is resolved correctly
maybe some strict rules was introduced in DBR 18.xx

Re: Databricks DBR 18.1 access workspace files error

Ashwin_DSA — Tue, 17 Mar 2026 14:42:35 GMT

Hi @Youry,

Thank you for sharing this. I will try to flag it internally, but I highly recommend that you raise a support ticket to ensure it gets prioritised.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

Re: Databricks DBR 18.1 access workspace files error

der — Tue, 17 Mar 2026 15:19:36 GMT

I will try this code next time if it happens. Keep you updated

Re: Databricks DBR 18.1 access workspace files error

Ashwin_DSA — Tue, 17 Mar 2026 16:04:06 GMT

@der, @Youry - FYI. I've now flagged this internally as well.

Re: Databricks DBR 18.1 access workspace files error

Youry — Wed, 18 Mar 2026 07:04:41 GMT

Thanks !
however if the only documented option is to use Workspace Files API then probably have to adjust code accordingly. FYI We're using Workspace/code/ as part of CI/CD process - code and configs as well are landed into subdirectory by pipeline. Moving part of it to DBFS adds complexity

Re: Databricks DBR 18.1 access workspace files error

der — Wed, 18 Mar 2026 09:48:36 GMT

Today I see the issue again on a dedicated cluster with DBR 18.1 with deactivated DBFS on workspace level

Accessing File with Python result into:

FileNotFoundError: [Errno 2] No such file or directory

Both subprocess run can access main folder "base" and the config.json "file"

dmesg command does not work (even with sudo)

sudo dmesg | tail -n 50 || true
dmesg: read kernel buffer failed: Operation not permitted

dbutils works too

dbutils.fs.ls("file:/Workspace/Users/<USER>")

and also native listing of the directory works

import os
files = os.listdir("/Workspace/Users/<USER>")

So it looks for me like there is something wrong or not working in /databricks/python/lib/python3.12/site-packages/IPython/core/interactiveshell.py with workspace files

File /databricks/python/lib/python3.12/site-packages/IPython/core/interactiveshell.py:324, in _modified_open(file, *args, **kwargs) 317 if file in {0, 1, 2}: 318 raise ValueError( 319 f"IPython won't let you open fd={file} by default " 320 "as it is likely to crash IPython. If you know what you are doing, " 321 "you can use builtins' open." 322 ) --> 324 return io_open(file, *args, **kwargs)

Re: Databricks DBR 18.1 access workspace files error

Ashwin_DSA — Wed, 18 Mar 2026 19:20:56 GMT

Hi @Youry,

Understood. Did you happen to raise a support ticket? If so, could you please share the reference number? It will help me to escalate the issue.

Re: Databricks DBR 18.1 access workspace files error

der — Mon, 23 Mar 2026 07:31:32 GMT

Hi @Ashwin_DSA

I created a support ticket and will provide you the reference number as soon as i get it.

Re: Databricks DBR 18.1 access workspace files error

Malthe — Mon, 23 Mar 2026 09:39:44 GMT

This happens to us on 18.0.3, but I haven't seen it on < 18.