"with open" not working in single user access mode cluster (no such file or directory found)

[rizkyjarr]

Hi fellow engineers,

So i was trying to read binary files (.jpg) in a ADLS2 mounted container

But when im trying to read the file using "with open" i kept getting an error: No such file or directory found

I've read something related to this matter on Solved: "with open" not working with Shared Access Cluster... - Databricks Community - 54653 that the cluster may be in a shared mode, but my current cluster is using a single-user mode but still keep getting error: 

Is there anyone here who solved this problem? I'd appreciate any advice and solutions for this matter. Thank you!

[UmaMahesh1]

Weird...
I'm able to access it without any issues. 

In case you are using community edition clusters, try copying the file to the driver node first and then read it.

2nd option is below:

[rizkyjarr]

Hi @UmaMahesh1 , thanks for your reply.

Yeah it is so weird and i can't seem to find the answer for that still (maybe that im using a community edition?, since i created databricks resource from my azure trial subscription)

Anyway, i wanted to clarify on these points

• "In case you are using community edition clusters, try copying the file to the driver node first and then read it" . Does that i mean i have to copy and write the file into databricks local file storage first? (idk what's the term for this, i suppose its like /temp /tmp storage?)
• 2nd option - i think i've tried reading using dataframe before, but it seemed it just read the binary string but couldn't write it back to binary files? CMIIW

Looking forward for your advice on this hehe but at your convenience, thanks