cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

i am trying to read csv file using databricks, i am getting error like ......FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/tables/world_bank.csv'

Venky
New Contributor III

i am trying to read csv file using databricks, i am getting error like ......FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/tables/world_bank.csv'

image

19 REPLIES 19

Hubert-Dudek
Esteemed Contributor III

I see that you are using databricks-course-cluster which have probably some limited functionality. Not sure where dbfs is mounted there. When you are using dbutils it display path for dbfs mount (dbfs file system).

Please use spark code instead of pandas so it will be executed properly:

df = spark.read.csv('dbfs:/FileStore/tables/world_bank.csv')
display(df)

Alexis
New Contributor III

ops I didn't see the other answers, anyway here you have how to use %fs magic to do the same that dbutils.fs.ls() utils.

Just before to create the spark data frame, check if the file exists in the mentioned path.

You can use the %fs magic like this:

fs_magic

klllmmm
New Contributor II

Pls help,

I have the same problem

image

-werners-
Esteemed Contributor III

I see you use pandas to read from dbfs.

But pandas will only read from local files,

see this topic also. It is about databricks-connect but the same principles apply.

So what you should do is first read the file using spark.read.csv and then converting the spark df to a pandas df.

Alexis
New Contributor III

Hi

you can try:

my_df = spark.read.format("csv")

      .option("inferSchema","true")  # to get the types from your data

      .option("sep",",")            # if your file is using "," as separator

      .option("header","true")       # if your file have the header in the first row

      .load("/FileStore/tables/CREDIT_1.CSV")

display(my_df)

from above you can see that my_df is a spark dataframe and from there you can start with you code.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!