cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Unable to open files with python, but filesystem shows files exist

snoeprol
New Contributor II

Dear community,

I have the following problem:

%fs mv '/FileStore/Tree_point_classification-1.dlpk'  '/dbfs/mnt/group22/Tree_point_classification-1.dlpk'

I have uploaded a file of a ML-model and have transferred it to the directory with

When I now check for the file I get:

%fs ls '/dbfs/mnt/group22/'
 
path	name	size
dbfs:/dbfs/mnt/group22/1_2_with_color.las	1_2_with_color.las	36439
dbfs:/dbfs/mnt/group22/Tree_point_classification-1.dlpk	Tree_point_classification-1.dlpk	14517532
dbfs:/dbfs/mnt/group22/out.laz	out.laz	39092572

Meaning the file is there. But now whenever I try

with open('/dbfs/mnt/group22/Tree_point_classification-1.dlpk') as f:
                  pass

it gives me a FileNotFoundException. Why can the filesystem access the file but my Python interpreter can not?

Thanks in advance

1 ACCEPTED SOLUTION

Accepted Solutions

Hubert-Dudek
Esteemed Contributor III

There is dbfs:/dbfs/ displayed maybe file is in /dbfs/dbfs directory? Please check it and try to open with open('/dbfs/dbfs. You can also use "data" from left menu to check what is in dbfs file system more easily.

View solution in original post

4 REPLIES 4

Kaniz
Community Manager
Community Manager

Hi @ snoeprol! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will get back to you soon. Thanks.

-werners-
Esteemed Contributor III

your code is correct. Things that come to mind are:

  • typo in the filename?
  • not enough permissions on the mount to open the file

Can you check this outside of Databricks, to see if the file is actually there, with that name AND correct permissions?

Hubert-Dudek
Esteemed Contributor III

There is dbfs:/dbfs/ displayed maybe file is in /dbfs/dbfs directory? Please check it and try to open with open('/dbfs/dbfs. You can also use "data" from left menu to check what is in dbfs file system more easily.

Yep this was it. I found the solution yesterday, and tried to respond, but it didn't post on my phone. I should move files to

mnt/ .....

Instead of

/dbfs/mnt/....

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.