topic Re: How to read a file in pandas in a databricks environment? in Data Engineering

How to read a file in pandas in a databricks environment?

tinendra — Thu, 16 Dec 2021 04:28:36 GMT

Hi,

When I was trying to read the CSV files using pandas I am getting an error which I have mentioned below.

df=pd.read_csv("/dbfs/FileStore/tables/badrecord-1.csv")

Error: FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/tables/badrecord-1.csv'

df=pd.read_csv("/FileStore/tables/badrecord-1.csv")

Error: FileNotFoundError: [Errno 2] No such file or directory: '/FileStore/tables/badrecord-1.csv'

Anyone, please tell me how to resolve the above error. thanks

Re: How to read a file in pandas in a databricks environment?

Anonymous — Fri, 17 Dec 2021 16:55:11 GMT

So pandas runs on the driver and will read from the driver's filesystem. DBFS is the distributed file system. You can use %sh fs ls to explore the files on the driver.

Re: How to read a file in pandas in a databricks environment?

tinendra — Fri, 28 Oct 2022 13:52:22 GMT

dbutils.fs.ls("/FileStore/tables/badrecord-1.csv")

the above file is there in that particular location but still getting the same error