04-12-2022 05:52 AM
Hi Can you help me why Pandas code not working..but Pyspark is working..
import pandas as pd
pdf = pd.read_csv('/FileStore/tables/new.csv',sep=',')
Error : No such file exists...
below is worked..
df = spark.read.csv("/FileStore/tables/new.csv", sep=",", header='True')
04-12-2022 08:41 AM
It might has to do with the path as @Hubert Dudek already mentioned:
df = spark.read.csv("dbfs:/FileStore/tables/new.csv", sep=",", header='True')
04-12-2022 05:55 AM
Try to add /dbfs/ or dbfs: prefix
04-12-2022 06:14 AM
Yeah..I tried all options...still no file exists..
So I am converting Py spark DF to Pandas DF...
I am interested to know why below is not working..
pdf = pd.read_csv('/FileStore/tables/new.csv',sep=',')
04-12-2022 08:41 AM
It might has to do with the path as @Hubert Dudek already mentioned:
df = spark.read.csv("dbfs:/FileStore/tables/new.csv", sep=",", header='True')
04-13-2022 01:48 AM
Hi @Rafael Rockenbach and @Hubert Dudek , It was so nice to have your response. Thank you for the time you put into our community. I really want you to know how much we appreciate that.
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.