What are some guidelines for migrating to DBR 7/Spark 3?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-07-2021 03:57 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-07-2021 03:57 PM
- Make sure scala version is 2.12. Also if the library is based on Spark 2.x then update that library to Spark 2.4.5 for Scala 2.12 support.
- For SBT projects use crossScalaVersion setting to build libraries for both versions of Scala
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-02-2021 07:11 PM
Please refer to the below reference for switching to DBR 7.x
We have extended our DBR 6.4 support until December 2021, DBR 6.4 extended support - Release notes: https://docs.databricks.com/release-notes/runtime/6.4x.html
Migration guide to DBR 7.x:
https://docs.databricks.com/release-notes/runtime/7.x-migration.html https://spark.apache.org/docs/latest/sql-migration-guide.html
New Features and Changes in Spark 3 (DBR 7.x)
DBR 7.x AQE: https://databricks.com/blog/2020/05/29/adaptive-query-execution-speeding-up-spark-sql-at-runtime.htm...
New changes to Dates and Timestamp: https://databricks.com/blog/2020/07/22/a-comprehensive-look-at-dates-and-timestamps-in-apache-spark-...
New changes to Pandas UDFs and Python Type hints: https://databricks.com/blog/2020/05/20/new-pandas-udfs-and-python-type-hints-in-the-upcoming-release...
Get an overview on DBR 7.3 LTS: https://databricks.com/blog/2021/03/08/upgrade-production-workloads-to-be-safer-easier-and-faster-wi...