cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

How to read a file in pandas in a databricks environment?

tinendra
New Contributor III

Hi,

When I was trying to read the CSV files using pandas I am getting an error which I have mentioned below.

df=pd.read_csv("/dbfs/FileStore/tables/badrecord-1.csv")

Error: FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/tables/badrecord-1.csv'

df=pd.read_csv("/FileStore/tables/badrecord-1.csv")

Error: FileNotFoundError: [Errno 2] No such file or directory: '/FileStore/tables/badrecord-1.csv'

Anyone, please tell me how to resolve the above error. thanks

3 REPLIES 3

Kaniz
Community Manager
Community Manager

Hi @Tinendra Kumar​ , Can you please check the exact location of the file via this code?

dbutils.fs.ls("/FileStore/tables/badrecord-1.csv")

Anonymous
Not applicable

So pandas runs on the driver and will read from the driver's filesystem. DBFS is the distributed file system. You can use %sh fs ls to explore the files on the driver.

tinendra
New Contributor III

dbutils.fs.ls("/FileStore/tables/badrecord-1.csv")

the above file is there in that particular location but still getting the same error

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.