topic Re: Problems with pandas.read_parquet() and path in Data Engineering

Problems with pandas.read_parquet() and path

johnb1 — Wed, 30 Nov 2022 19:20:52 GMT

I am doing the "Data Engineering with Databricks V2" learning path.

I cannot run "DE 4.2 - Providing Options for External Sources", as the first code cell does not run successful:

%run ../Includes/Classroom-Setup-04.2

Screenshot 1:

Inside the setup notebook, the code crashes at the following command (see screenshot 2):

df = pd.read_parquet(path = datasource_path.replace("dbfs:/", '/dbfs/'))

The error message is:

FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/mnt/dbacademy-datasets/data-engineering-with-databricks/v02/ecommerce/raw/users-historical'

Screenshot 2:

There seems to be an issue with the path, even though it actually exists:

Screenshot 3:

I played around a little with the path specification, but nothing helped:

Screenshot 4:

Re: Problems with pandas.read_parquet() and path

UmaMahesh1 — Wed, 30 Nov 2022 19:42:30 GMT

Hi @John B

Can you please try by removing the dbfs and starting with /mnt only.

Also, if this does not work, can you please upload that notebooks DBC archive, so that I would be able to check the details.

Cheers..

Re: Problems with pandas.read_parquet() and path

UmaMahesh1 — Wed, 30 Nov 2022 19:54:46 GMT

Also @John B

Assuming this is an old training course, check the same using a community cluster with DBR version less than 7. Some old training courses mount points are disabled in DBR 7+.

Cheers...

Re: Problems with pandas.read_parquet() and path

UmaMahesh1 — Sat, 03 Dec 2022 08:22:16 GMT

@John B

Did your issue get resolved?

If not through the above methods, do ping the fix you did.

Cheers..

Re: Problems with pandas.read_parquet() and path

SS2 — Sun, 04 Dec 2022 03:14:32 GMT

Can u try like this.spark.read.parquet("dbfs:/mnt/.......")

Re: Problems with pandas.read_parquet() and path

johnb1 — Fri, 16 Dec 2022 11:35:43 GMT

Hi!

I can only use Runtime 7.3, 9.1., ..., 12.0. Minimum is 7.3. I am using DBR commnunity edition.

Br.

Re: Problems with pandas.read_parquet() and path

johnb1 — Fri, 16 Dec 2022 11:47:49 GMT

Hi @Uma Maheswara Rao Desula

Removing the dbfs and starting with /mnt only does not help.

Br.

Re: Problems with pandas.read_parquet() and path

johnb1 — Fri, 16 Dec 2022 11:58:18 GMT

Hi @S S

Reading in the file was successful. However, I got a pyspark.sql.dataframe.DataFrame object. This is not the same as a pandas DataFrame, right?

Br.

Re: Problems with pandas.read_parquet() and path

johnb1 — Fri, 16 Dec 2022 12:33:49 GMT

@Uma Maheswara Rao Desula I solved the issue using ss2's suggestion (see below). After reading in a Spark DataFrame I converted it into a pandas DataFrame using the ToPandas() method.

Re: Problems with pandas.read_parquet() and path

Aviral-Bhardwaj — Fri, 16 Dec 2022 15:04:30 GMT

Hey @S S ,

I can understand your issue

so to solve this import that DBC file and instead of question one there will be a folder for all solutions so explore solution one it will work.

Please upvote if you got some hint from my answer

Thanks

Aviral Bhardwaj

Re: Problems with pandas.read_parquet() and path

smkazim — Wed, 29 Mar 2023 15:11:08 GMT

Hello All,

I am getting the exact issue as motioned in the first pot here. I have tried all the solutions listed: -

Changing DBR to 7.3: Gave other errors related to libraries not present in that DBR version
Using spark.read.parquet: This is giving "AnalysisException: Unable to infer schema for Parquet. It must be specified manually." error. I have checked the parquet files exists in that location and they are not empty.
Exploring solutions folder: It is giving the same errors.

Any ideas what else I can try please.

Thanks.

Re: Problems with pandas.read_parquet() and path

vijaykumar99535 — Thu, 04 Jan 2024 09:35:45 GMT

I used spark.read.parquet and then convereted that to pandas dataframe and it worked for me.

Upvote if it helped you.

Re: Problems with pandas.read_parquet() and path

jonathanchcc — Fri, 02 Feb 2024 14:34:03 GMT

Thanks for sharing this helped me too 🤖

Re: Problems with pandas.read_parquet() and path

mrb_cookiebaker — Fri, 28 Jun 2024 14:37:20 GMT

Worked for me too! Thanks

Re: Problems with pandas.read_parquet() and path

Dibs — Fri, 30 Aug 2024 01:38:38 GMT

Thanks it helped.

Re: Problems with pandas.read_parquet() and path

vigneshmayil — Wed, 16 Oct 2024 15:35:13 GMT

spark solution worked

instead of

df = pd.read_parquet(path = datasource_path.replace("dbfs:/", '/dbfs/'))

I used

spark_df = spark.read.parquet("dbfs:/mnt/dbacademy-datasets/data-engineer-learning-path/v04/ecommerce/raw/users-historical/")
# Convert to pandas
DataFrame df = spark_df.toPandas()

Re: Problems with pandas.read_parquet() and path

hebied — Fri, 25 Oct 2024 06:45:53 GMT

Thanks for sharing bro ..It really helped.