"with open" not working in single user access mode cluster (no such file or directory found)

rizkyjarr
New Contributor II

‎06-19-2025 08:38 PM

Hi fellow engineers,

So i was trying to read binary files (.jpg) in a ADLS2 mounted container

But when im trying to read the file using "with open" i kept getting an error: No such file or directory found

rizkyjarr_0-1750390374120.png

I've read something related to this matter on Solved: "with open" not working with Shared Access Cluster... - Databricks Community - 54653 that the cluster may be in a shared mode, but my current cluster is using a single-user mode but still keep getting error: 

rizkyjarr_1-1750390546193.png

Is there anyone here who solved this problem? I'd appreciate any advice and solutions for this matter. Thank you!

0 Kudos

UmaMahesh1
Honored Contributor III

‎06-20-2025 07:43 AM

Weird...
I'm able to access it without any issues. 

UmaMahesh1_0-1750430283962.png

In case you are using community edition clusters, try copying the file to the driver node first and then read it.

2nd option is below:

UmaMahesh1_1-1750430566638.png

 

Uma Mahesh D
0 Kudos

rizkyjarr
New Contributor II
In response to UmaMahesh1

‎06-22-2025 07:14 PM

Hi @UmaMahesh1 , thanks for your reply.

Yeah it is so weird and i can't seem to find the answer for that still (maybe that im using a community edition?, since i created databricks resource from my azure trial subscription)

Anyway, i wanted to clarify on these points

  • "In case you are using community edition clusters, try copying the file to the driver node first and then read it" . Does that i mean i have to copy and write the file into databricks local file storage first? (idk what's the term for this, i suppose its like /temp /tmp storage?)
  • 2nd option - i think i've tried reading using dataframe before, but it seemed it just read the binary string but couldn't write it back to binary files? CMIIW

Looking forward for your advice on this hehe but at your convenience, thanks

0 Kudos

amenon
Databricks Employee
Databricks Employee

‎07-28-2025 03:10 AM

@rizkyjarr , did you run into the issue with `with open()` using `\dbfs\mnt` paths , while using a non-Unity catalog enabled workspace despite using the single user access mode cluster  as you pointed out?

0 Kudos