topic Re: Error as no such file when reading CSV file using pandas in Data Engineering

Error as no such file when reading CSV file using pandas

klllmmm — Tue, 24 May 2022 16:22:22 GMT

I'm trying to read a CSV file saved in data using pandas read_csv function. But it gives No such file error.

%fs
ls /FileStore/tables/
 
df= pd.read_csv('/dbfs/FileStore/tables/CREDIT_1.CSV')
 
 
df= pd.read_csv('/dbfs:/FileStore/tables/CREDIT_1.CSV')
 
 
df= pd.read_csv('/FileStore/tables/CREDIT_1.CSV')

Can someone pls help with this?

Re: Error as no such file when reading CSV file using pandas

-werners- — Wed, 25 May 2022 07:05:09 GMT

I see you use pandas to read from dbfs.

But pandas will only read from local files,

see this topic also. It is about databricks-connect but the same principles apply.

So what you should do is first read the file using spark.read.csv and then converting the spark df to a pandas df.

Re: Error as no such file when reading CSV file using pandas

Anonymous — Wed, 01 Jun 2022 07:01:12 GMT

Hi @Kelum Perera following - up did you get a chance to check @Werner Stinckens previous comments or do you need any further help on this?

Re: Error as no such file when reading CSV file using pandas

klllmmm — Fri, 03 Jun 2022 16:33:44 GMT

Thanks to @Werner Stinckens for the answer.

I understood that I have to use spark to read data from clusters.