topic Re: issue when reading csv in pandas in Get Started Discussions

issue when reading csv in pandas

sbs — Thu, 28 Mar 2024 13:34:57 GMT

Hello Team,

I've encountered an issue while attempting to read a CSV data file into a pandas DataFrame by uploading it into DBFS in the community version of Databricks. Below is the error I encountered along with the code snippet I used:

import pandas as pd

df1 = pd.read_csv("/dbfs/FileStore/shared_uploads/shiv/Dea.csv")

Error Encountered:
FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/shared_uploads/shared_uploads/shiv/Dea.csv'

However, when checking for the file using the dbutils.fs.ls command, I can see the file present:

dbutils.fs.ls("/FileStore/shared_uploads/shiv/Dea.csv")

Output:

[FileInfo(path='dbfs:/FileStore/shared_uploads/shiv/Dea.csv', name='Dea.csv', size=18679559, modificationTime=1711631849000)]

Re: issue when reading csv in pandas

Lakshay — Thu, 28 Mar 2024 17:59:34 GMT

Are you able to read file from that location using spark code?

Re: issue when reading csv in pandas

sbs — Thu, 28 Mar 2024 22:27:41 GMT

yes

Re: issue when reading csv in pandas

YuliyanBogdanov — Fri, 29 Mar 2024 10:25:50 GMT

Assuming dbutils.fs.ls works without the "dbfs:/" prefix, try using it directly, i.e. df1 = pd.read_csv("/FileStore/shared_uploads/shiv/Dea.csv") . Alternatively, adjust the path as needed if using a local file path df1 = pd.read_csv("dbfs:/FileStore/shared_uploads/shiv/Dea.csv"). Hope that helps.