topic Re: Pandas finds parquet file, Spark does not in Data Engineering

Pandas finds parquet file, Spark does not

JonW — Fri, 29 Dec 2023 20:02:57 GMT

I am having an issue with Databricks (Community Edition) where I can use Pandas to read a parquet file into a dataframe, but when I use Spark it states the file doesn't exist. I have tried reformatting the file path for spark but I can't seem to find a format that it will accept.

Any ideas?

Pandas:

parquet_file_path = "/dbfs/green_tripdata_2022-02.parquet" df = pd.read_parquet(parquet_file_path, engine='pyarrow') display(df)

Result:

Spark:

parquet_file_path = "/dbfs/green_tripdata_2022-02.parquet" df = spark.read.parquet(parquet_file_path) df.show()

Result:
AnalysisException: [PATH_NOT_FOUND] Path does not exist: dbfs:/dbfs/green_tripdata_2022-02.parquet.

Re: Pandas finds parquet file, Spark does not

Wojciech_BUK — Fri, 29 Dec 2023 23:02:43 GMT

Can you check those 3 options ? I don't remember which one will work and can't test it now, but I am sore one or two of those will work 🙂

parquet_file_path = "/green_tripdata_2022-02.parquet"

parquet_file_path = "green_tripdata_2022-02.parquet"

parquet_file_path = "dbfs:/green_tripdata_2022-02.parquet"

Re: Pandas finds parquet file, Spark does not

jose_gonzalez — Wed, 03 Jan 2024 22:30:23 GMT

Are you getting any error messages? what happens when you do a "ls /dbfs/"? are you able to list all the parquet files?