Databricks

dbu_spark · 10-20-2021

I have databricks runtime for a job set to latest 10.0 Beta (includes Apache Spark 3.2.0, Scala 2.12) .In the notebook when I check for the spark version, I see version 3.1.0 instead of version 3.2.0I need the Spark version 3.2 to process workloads a...

dbu_spark · 10-20-2021

I do not think it is loading Spark 3.2. I am still seeing the issue with writeUTF which has been fixed in Spark 3.2 -> https://github.com/apache/spark/pull/32788Caused by: java.io.UTFDataFormatException: encoded string too long: 97548 bytes at java....

dbu_spark · 10-20-2021

Thanks for the update @Dan Zafar . Just ran the job again and still seeing spark version 3.1.0. Should I be using spark32 (or something similar) when invoking spark session for me to pick up the correct spark version?Any ETA on the spark 3.2 version...

dbu_spark · 10-20-2021

I am not using the pool. Thanks for the update though. Hopefully this gets fixed soon.

Databricks

User Stats

User Activity

Older Spark Version loaded into the spark notebook

Re: Older Spark Version loaded into the spark notebook

Re: Older Spark Version loaded into the spark notebook

Re: Older Spark Version loaded into the spark notebook