topic Re: "with open" not working in single user access mode cluster (no such file or directory in Data Engineering

"with open" not working in single user access mode cluster (no such file or directory found)

rizkyjarr — Fri, 20 Jun 2025 03:38:35 GMT

Hi fellow engineers,

So i was trying to read binary files (.jpg) in a ADLS2 mounted container

But when im trying to read the file using "with open" i kept getting an error: No such file or directory found

I've read something related to this matter on Solved: "with open" not working with Shared Access Cluster... - Databricks Community - 54653 that the cluster may be in a shared mode, but my current cluster is using a single-user mode but still keep getting error:

Is there anyone here who solved this problem? I'd appreciate any advice and solutions for this matter. Thank you!

Re: "with open" not working in single user access mode cluster (no such file or directory

UmaMahesh1 — Fri, 20 Jun 2025 14:43:16 GMT

Weird...
I'm able to access it without any issues.

In case you are using community edition clusters, try copying the file to the driver node first and then read it.

2nd option is below:

Re: "with open" not working in single user access mode cluster (no such file or directory

rizkyjarr — Mon, 23 Jun 2025 02:14:02 GMT

Hi @UmaMahesh1 , thanks for your reply.

Yeah it is so weird and i can't seem to find the answer for that still (maybe that im using a community edition?, since i created databricks resource from my azure trial subscription)

Anyway, i wanted to clarify on these points

"In case you are using community edition clusters, try copying the file to the driver node first and then read it" . Does that i mean i have to copy and write the file into databricks local file storage first? (idk what's the term for this, i suppose its like /temp /tmp storage?)
2nd option - i think i've tried reading using dataframe before, but it seemed it just read the binary string but couldn't write it back to binary files? CMIIW

Looking forward for your advice on this hehe but at your convenience, thanks

Re: "with open" not working in single user access mode cluster (no such file or directory

amenon — Mon, 28 Jul 2025 10:10:38 GMT

@rizkyjarr , did you run into the issue with `with open()` using `\dbfs\mnt` paths , while using a non-Unity catalog enabled workspace despite using the single user access mode cluster as you pointed out?