What are some guidelines for migrating to DBR 7/Spark 3?

[User16789201666]

What are some guidelines for migrating to DBR 7/Spark 3?

[User16789201666]

• Make sure scala version is 2.12. Also if the library is based on Spark 2.x then update that library to Spark 2.4.5 for Scala 2.12 support.
• For SBT projects use crossScalaVersion setting to build libraries for both versions of Scala

[shan_chandra]

Please refer to the below reference for switching to DBR 7.x

We have extended our DBR 6.4 support until December 2021, DBR 6.4 extended support - Release notes: https://docs.databricks.com/release-notes/runtime/6.4x.html

Migration guide to DBR 7.x:

https://docs.databricks.com/release-notes/runtime/7.x-migration.html https://spark.apache.org/docs/latest/sql-migration-guide.html

New Features and Changes in Spark 3 (DBR 7.x)

DBR 7.x AQE: https://databricks.com/blog/2020/05/29/adaptive-query-execution-speeding-up-spark-sql-at-runtime.htm...

New changes to Dates and Timestamp: https://databricks.com/blog/2020/07/22/a-comprehensive-look-at-dates-and-timestamps-in-apache-spark-...

New changes to Pandas UDFs and Python Type hints: https://databricks.com/blog/2020/05/20/new-pandas-udfs-and-python-type-hints-in-the-upcoming-release...

Get an overview on DBR 7.3 LTS: https://databricks.com/blog/2021/03/08/upgrade-production-workloads-to-be-safer-easier-and-faster-wi...