topic Re: Older Spark Version loaded into the spark notebook in Data Engineering

Older Spark Version loaded into the spark notebook

dbu_spark — Wed, 20 Oct 2021 15:47:54 GMT

I have databricks runtime for a job set to latest 10.0 Beta (includes Apache Spark 3.2.0, Scala 2.12) .

In the notebook when I check for the spark version, I see version 3.1.0 instead of version 3.2.0

I need the Spark version 3.2 to process workloads as that version has the fix for https://github.com/apache/spark/pull/32788

Screenshot with the cluster configuration and the older spark version in notebook attached.

Re: Older Spark Version loaded into the spark notebook

Hubert-Dudek — Wed, 20 Oct 2021 16:11:02 GMT

If you use pool please check what preloaded version is set also in pool.

If it is not that problem I can not help as I even don't see yet 10.0 (and after all it is Beta)

Re: Older Spark Version loaded into the spark notebook

Anonymous — Wed, 20 Oct 2021 16:27:56 GMT

I got the same thing when I tested it out. I guess that's why it's Beta, should get fixed soon I imagine.

Re: Older Spark Version loaded into the spark notebook

dbu_spark — Wed, 20 Oct 2021 16:54:41 GMT

I am not using the pool. Thanks for the update though. Hopefully this gets fixed soon.

Re: Older Spark Version loaded into the spark notebook

Dan_Z — Wed, 20 Oct 2021 18:17:27 GMT

This is due to some legalese related to Open-Source Spark and Databricks' Spark. Since Open-Source Spark has not released v3.2 yet, we are not allowed to call the one on DBR 10 v3.2 yet. It may have that patch already in there though.

Re: Older Spark Version loaded into the spark notebook

dbu_spark — Wed, 20 Oct 2021 18:45:26 GMT

Thanks for the update @Dan Zafar .

Just ran the job again and still seeing spark version 3.1.0. Should I be using spark32 (or something similar) when invoking spark session for me to pick up the correct spark version?

Any ETA on the spark 3.2 version availability will be great.

Thanks

Re: Older Spark Version loaded into the spark notebook

Dan_Z — Wed, 20 Oct 2021 19:11:51 GMT

It should have all the features you need. Check it out. Legally we can't call it Spark 3.2 yet.

Re: Older Spark Version loaded into the spark notebook

dbu_spark — Wed, 20 Oct 2021 19:25:10 GMT

I do not think it is loading Spark 3.2. I am still seeing the issue with writeUTF which has been fixed in Spark 3.2 -> https://github.com/apache/spark/pull/32788

Caused by: java.io.UTFDataFormatException: encoded string too long: 97548 bytes
	at java.io.DataOutputStream.writeUTF(DataOutputStream.java:364)
	at java.io.DataOutputStream.writeUTF(DataOutputStream.java:323)

Anyways, I will wait for the databricks runtime to correctly reflect the correct version.

Re: Older Spark Version loaded into the spark notebook

Dan_Z — Wed, 20 Oct 2021 21:44:07 GMT

Yes- this version probably has the Databricks internal features slated for Spark 3.2, but the features/patches contributed by the open-source community may still be coming. Sorry this isn't available yet. I'm sure it will be very soon. Happy coding!

Re: Older Spark Version loaded into the spark notebook

-werners- — Thu, 21 Oct 2021 12:32:23 GMT

I just noticed that (on Azure anyway) 10.0 is NOT in beta anymore.

So 'very soon' was indeed very soon.

Re: Older Spark Version loaded into the spark notebook

jose_gonzalez — Thu, 21 Oct 2021 16:47:45 GMT

hi @Dhaivat Upadhyay ,

Good news, DBR 10 was release yesterday October 20th. You can find more details in the release notes website