topic Py Spark Pandas Code diff in Data Engineering

Py Spark Pandas Code diff

Krishscientist — Tue, 12 Apr 2022 12:52:43 GMT

Hi Can you help me why Pandas code not working..but Pyspark is working..

import pandas as pd

pdf = pd.read_csv('/FileStore/tables/new.csv',sep=',')

Error : No such file exists...

below is worked..

df = spark.read.csv("/FileStore/tables/new.csv", sep=",", header='True')

Hubert-Dudek — Tue, 12 Apr 2022 12:55:14 GMT

Try to add /dbfs/ or dbfs: prefix

Krishscientist — Tue, 12 Apr 2022 13:14:57 GMT

Yeah..I tried all options...still no file exists..

So I am converting Py spark DF to Pandas DF...

I am interested to know why below is not working..

pdf = pd.read_csv('/FileStore/tables/new.csv',sep=',')

RRO — Tue, 12 Apr 2022 15:41:17 GMT

It might has to do with the path as @Hubert Dudek already mentioned:

df = spark.read.csv("dbfs:/FileStore/tables/new.csv", sep=",", header='True')