cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Unable to open files with python, but filesystem shows files exist

snoeprol
New Contributor II

Dear community,

I have the following problem:

%fs mv '/FileStore/Tree_point_classification-1.dlpk'  '/dbfs/mnt/group22/Tree_point_classification-1.dlpk'

I have uploaded a file of a ML-model and have transferred it to the directory with

When I now check for the file I get:

%fs ls '/dbfs/mnt/group22/'
 
path	name	size
dbfs:/dbfs/mnt/group22/1_2_with_color.las	1_2_with_color.las	36439
dbfs:/dbfs/mnt/group22/Tree_point_classification-1.dlpk	Tree_point_classification-1.dlpk	14517532
dbfs:/dbfs/mnt/group22/out.laz	out.laz	39092572

Meaning the file is there. But now whenever I try

with open('/dbfs/mnt/group22/Tree_point_classification-1.dlpk') as f:
                  pass

it gives me a FileNotFoundException. Why can the filesystem access the file but my Python interpreter can not?

Thanks in advance

1 ACCEPTED SOLUTION

Accepted Solutions

Hubert-Dudek
Esteemed Contributor III

There is dbfs:/dbfs/ displayed maybe file is in /dbfs/dbfs directory? Please check it and try to open with open('/dbfs/dbfs. You can also use "data" from left menu to check what is in dbfs file system more easily.

View solution in original post

3 REPLIES 3

-werners-
Esteemed Contributor III

your code is correct. Things that come to mind are:

  • typo in the filename?
  • not enough permissions on the mount to open the file

Can you check this outside of Databricks, to see if the file is actually there, with that name AND correct permissions?

Hubert-Dudek
Esteemed Contributor III

There is dbfs:/dbfs/ displayed maybe file is in /dbfs/dbfs directory? Please check it and try to open with open('/dbfs/dbfs. You can also use "data" from left menu to check what is in dbfs file system more easily.

Yep this was it. I found the solution yesterday, and tried to respond, but it didn't post on my phone. I should move files to

mnt/ .....

Instead of

/dbfs/mnt/....

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group