topic Re: Unable to open files with python, but filesystem shows files exist in Data Engineering

Unable to open files with python, but filesystem shows files exist

snoeprol — Sun, 17 Oct 2021 12:25:45 GMT

Dear community,

I have the following problem:

%fs mv '/FileStore/Tree_point_classification-1.dlpk'  '/dbfs/mnt/group22/Tree_point_classification-1.dlpk'

I have uploaded a file of a ML-model and have transferred it to the directory with

When I now check for the file I get:

%fs ls '/dbfs/mnt/group22/'
 
path	name	size
dbfs:/dbfs/mnt/group22/1_2_with_color.las	1_2_with_color.las	36439
dbfs:/dbfs/mnt/group22/Tree_point_classification-1.dlpk	Tree_point_classification-1.dlpk	14517532
dbfs:/dbfs/mnt/group22/out.laz	out.laz	39092572

Meaning the file is there. But now whenever I try

with open('/dbfs/mnt/group22/Tree_point_classification-1.dlpk') as f:
                  pass

it gives me a FileNotFoundException. Why can the filesystem access the file but my Python interpreter can not?

Thanks in advance

Re: Unable to open files with python, but filesystem shows files exist

-werners- — Mon, 18 Oct 2021 08:11:46 GMT

your code is correct. Things that come to mind are:

typo in the filename?
not enough permissions on the mount to open the file

Can you check this outside of Databricks, to see if the file is actually there, with that name AND correct permissions?

Re: Unable to open files with python, but filesystem shows files exist

Hubert-Dudek — Mon, 18 Oct 2021 10:38:36 GMT

There is dbfs:/dbfs/ displayed maybe file is in /dbfs/dbfs directory? Please check it and try to open with open('/dbfs/dbfs. You can also use "data" from left menu to check what is in dbfs file system more easily.

Re: Unable to open files with python, but filesystem shows files exist

snoeprol — Mon, 18 Oct 2021 11:08:40 GMT

Yep this was it. I found the solution yesterday, and tried to respond, but it didn't post on my phone. I should move files to

mnt/ .....

Instead of

/dbfs/mnt/....