cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Reading single file from Databricks DBFS

Saf4Databricks
New Contributor III

I have a Test.csv file in FileStore of DBFS in Databricks Community edition. When I try to read the file using With Open, I get the following error:FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/tables/Test.csv'

 

import os
with open('/dbfs/FileStore/tables/Test.csv') as testFile:
    testContent = testFile.read()

 

Question: What could be a cause of the error and how can we fix it?

The file gets loaded successfully in a df as follows:

 

df = spark.read.csv("/FileStore/tables/Test.csv", header=True, inferSchema=True)

 

2 REPLIES 2

EricRM
New Contributor II

From what I know, you can't acess filestore directly through python, it's something that is not supported by databricks and there is some reasons for it.
You should use instead the spark function as you mentioned or a db function(dbutils)

Saf4Databricks
New Contributor III

@EricRM It should work. Please see the accepted response from this same forum here. So, we still need to find a cause of the error. Following is the detailed error message. Maybe, this will help readers understand the issue better and help it resolve:

FileNotFoundError                         Traceback (most recent call last)
File <command-1762145911467087>, line 2
      1 import os
----> 2 with open('/dbfs/FileStore/tables/Test.csv', 'rb') as testFile:
      3     testContent = testFile.read()

File /databricks/python/lib/python3.11/site-packages/IPython/core/interactiveshell.py:286, in _modified_open(file, *args, **kwargs)
    279 if file in {0, 1, 2}:
    280     raise ValueError(
    281         f"IPython won't let you open fd={file} by default "
    282         "as it is likely to crash IPython. If you know what you are doing, "
    283         "you can use builtins' open."
    284     )
--> 286 return io_open(file, *args, **kwargs)

FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/tables/Test.csv'

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group