cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

i am trying to read csv file using databricks, i am getting error like ......FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/tables/world_bank.csv'

Venky
New Contributor III

i am trying to read csv file using databricks, i am getting error like ......FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/tables/world_bank.csv'

image

1 ACCEPTED SOLUTION

Accepted Solutions

Alexis
New Contributor III

Hi

you can try:

my_df = spark.read.format("csv")

      .option("inferSchema","true")  # to get the types from your data

      .option("sep",",")            # if your file is using "," as separator

      .option("header","true")       # if your file have the header in the first row

      .load("/FileStore/tables/CREDIT_1.CSV")

display(my_df)

from above you can see that my_df is a spark dataframe and from there you can start with you code.

View solution in original post

18 REPLIES 18

-werners-
Esteemed Contributor III

This means the path to the file you typed is not valid, the file is not there.

Can you check in the Data/DBFS page if the file is there (or via dbutils.fs.ls)?

Venky
New Contributor III

No, file path is correct

image

I am wondering how it worked for you. I have tried the same steps and no

luck. using dbutils i am finding file but when i read using pandas its saying no such file or directory exists.

dbutils.fs.ls("FileStore/shared_uploads/kumarpalle/Covid19Europedata.csv")

image 

Please help here. uploaded the same file that am working on.

Also, please check if you have applied any settings that we are missing.

Alexis
New Contributor III

Hi Kumar,

you're almost there, try this:

df1 = pd.read_csv("/dbfs/FileStore/shared_uploads/kumarpalle/Covid19Europedata.csv")

df1.describe()

Cheers!

Hi @Alexis​ ,

Thanks for quick response. It above method didn't worked but reading using spark and appending to Pandas it worked.

Here are the steps that i followed.

df1 = spark.read.format("csv").option("header", "true").load("dbfs:/FileStore/shared_uploads/kumarpalle/Covid19Europedata-1.csv").toPandas()

df1.head()

image 

@Venky D​ Please follow the above steps to read using Spark as pandas doesn't work through dbfs.  

same issue here... .toPandas() was also my only solution... otherwise error!

I did the same as you, but the reading does not work.

SanchesSan_0-1693410377818.png

Can you help me? thanks

 

Venky
New Contributor III

Please help me on this error

-werners-
Esteemed Contributor III

can you try without the dbfs part?

so /Filestore/tables/world_bank.csv

Venky
New Contributor III

it's not working

Hubert-Dudek
Esteemed Contributor III

I see that you are using databricks-course-cluster which have probably some limited functionality. Not sure where dbfs is mounted there. When you are using dbutils it display path for dbfs mount (dbfs file system).

Please use spark code instead of pandas so it will be executed properly:

df = spark.read.csv('dbfs:/FileStore/tables/world_bank.csv')
display(df)

Alexis
New Contributor III

ops I didn't see the other answers, anyway here you have how to use %fs magic to do the same that dbutils.fs.ls() utils.

Just before to create the spark data frame, check if the file exists in the mentioned path.

You can use the %fs magic like this:

fs_magic

klllmmm
New Contributor II

Pls help,

I have the same problem

image

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group