Reading single file from Databricks DBFS
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-05-2024 07:12 PM - edited 09-05-2024 07:22 PM
I have a Test.csv file in FileStore of DBFS in Databricks Community edition. When I try to read the file using With Open, I get the following error:FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/tables/Test.csv'
import os
with open('/dbfs/FileStore/tables/Test.csv') as testFile:
testContent = testFile.read()
Question: What could be a cause of the error and how can we fix it?
The file gets loaded successfully in a df as follows:
df = spark.read.csv("/FileStore/tables/Test.csv", header=True, inferSchema=True)
- Labels:
-
Spark
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-05-2024 07:31 PM
From what I know, you can't acess filestore directly through python, it's something that is not supported by databricks and there is some reasons for it.
You should use instead the spark function as you mentioned or a db function(dbutils)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-05-2024 08:19 PM - edited 09-05-2024 08:28 PM
@EricRM It should work. Please see the accepted response from this same forum here. So, we still need to find a cause of the error. Following is the detailed error message. Maybe, this will help readers understand the issue better and help it resolve:
FileNotFoundError Traceback (most recent call last)
File <command-1762145911467087>, line 2
1 import os
----> 2 with open('/dbfs/FileStore/tables/Test.csv', 'rb') as testFile:
3 testContent = testFile.read()
File /databricks/python/lib/python3.11/site-packages/IPython/core/interactiveshell.py:286, in _modified_open(file, *args, **kwargs)
279 if file in {0, 1, 2}:
280 raise ValueError(
281 f"IPython won't let you open fd={file} by default "
282 "as it is likely to crash IPython. If you know what you are doing, "
283 "you can use builtins' open."
284 )
--> 286 return io_open(file, *args, **kwargs)
FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/tables/Test.csv'

