Error as no such file when reading CSV file using pandas

[klllmmm]

I'm trying to read a CSV file saved in data using pandas read_csv function. But it gives No such file error.

%fs
ls /FileStore/tables/
 
df= pd.read_csv('/dbfs/FileStore/tables/CREDIT_1.CSV')
 
 
df= pd.read_csv('/dbfs:/FileStore/tables/CREDIT_1.CSV')
 
 
df= pd.read_csv('/FileStore/tables/CREDIT_1.CSV')

Can someone pls help with this?

[-werners-]

I see you use pandas to read from dbfs.

But pandas will only read from local files,

see this topic also. It is about databricks-connect but the same principles apply.

So what you should do is first read the file using spark.read.csv and then converting the spark df to a pandas df.

[Anonymous]

Hi @Kelum Perera​ following - up did you get a chance to check @Werner Stinckens​ previous comments or do you need any further help on this?

[klllmmm]

Thanks to @Werner Stinckens​ for the answer.

I understood that I have to use spark to read data from clusters.