topic Re: Reading single file from Databricks DBFS in Data Engineering

Reading single file from Databricks DBFS

Saf4Databricks — Fri, 06 Sep 2024 02:22:14 GMT

I have a Test.csv file in FileStore of DBFS in Databricks Community edition. When I try to read the file using With Open, I get the following error:FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/tables/Test.csv'

import os with open('/dbfs/FileStore/tables/Test.csv') as testFile: testContent = testFile.read()

Question: What could be a cause of the error and how can we fix it?

The file gets loaded successfully in a df as follows:

df = spark.read.csv("/FileStore/tables/Test.csv", header=True, inferSchema=True)

Re: Reading single file from Databricks DBFS

EricRM — Fri, 06 Sep 2024 02:31:48 GMT

From what I know, you can't acess filestore directly through python, it's something that is not supported by databricks and there is some reasons for it.
You should use instead the spark function as you mentioned or a db function(dbutils)

Re: Reading single file from Databricks DBFS

Saf4Databricks — Fri, 06 Sep 2024 03:28:21 GMT

@EricRM It should work. Please see the accepted response from this same forum here. So, we still need to find a cause of the error. Following is the detailed error message. Maybe, this will help readers understand the issue better and help it resolve:

FileNotFoundError Traceback (most recent call last) File <command-1762145911467087>, line 2 1 import os ----> 2 with open('/dbfs/FileStore/tables/Test.csv', 'rb') as testFile: 3 testContent = testFile.read() File /databricks/python/lib/python3.11/site-packages/IPython/core/interactiveshell.py:286, in _modified_open(file, *args, **kwargs) 279 if file in {0, 1, 2}: 280 raise ValueError( 281 f"IPython won't let you open fd={file} by default " 282 "as it is likely to crash IPython. If you know what you are doing, " 283 "you can use builtins' open." 284 ) --> 286 return io_open(file, *args, **kwargs) FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/tables/Test.csv'