topic Hive version after Upgrade Azure Databricks from 6.4 (Spark 2) to 9.1 (Spark 3) in Data Engineering

Hive version after Upgrade Azure Databricks from 6.4 (Spark 2) to 9.1 (Spark 3)

jeffreym9 — Thu, 16 Dec 2021 21:55:05 GMT

I have upgraded the Azure Databricks from 6.4 to 9.1 which enable me to use Spark3. As far as I know, the Hive version has to be upgraded to 2.3.7 as well as discussed in:

https://community.databricks.com/s/question/0D53f00001HKHy2CAH/how-to-upgrade-internal-hive-metadata-store-version

I have tried not given those options and continue using Hive 0.13 to run my Spark3 application and everything seems fine. Is it a must to upgrade the Hive metastore to 2.3.7 for Spark3 program to run? Cause everything seems find without adding:

spark.sql.hive.metastore.version 2.3.7
spark.sql.hive.metastore.jars builtin

Re: Hive version after Upgrade Azure Databricks from 6.4 (Spark 2) to 9.1 (Spark 3)

Atanu — Wed, 12 Jan 2022 02:55:03 GMT

@Jeffrey Mak all supported version is mention here https://docs.microsoft.com/en-us/azure/databricks/data/metastores/external-hive-metastore also, I do not think builtin will work.

For all other Hive versions, Azure Databricks recommends that you download the metastore JARs and set the configuration
spark.sql.hive.metastore.jars
to point to the downloaded JARs using the procedure described in Download the metastore jars and point to them.

Re: Hive version after Upgrade Azure Databricks from 6.4 (Spark 2) to 9.1 (Spark 3)

Anonymous — Wed, 26 Jan 2022 16:19:55 GMT

@Jeffrey Mak - Does Atanu's answer resolve the issue for you? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?

Re: Hive version after Upgrade Azure Databricks from 6.4 (Spark 2) to 9.1 (Spark 3)

jeffreym9 — Wed, 26 Jan 2022 19:09:04 GMT

I'm asking about Datatricks version 9.1. I've follow the url given (https://docs.microsoft.com/en-us/azure/databricks/data/metastores/external-hive-metastore). Do you mind letting me know where in the table is mentioning the supported hive version for Databricks 9.1?

Re: Hive version after Upgrade Azure Databricks from 6.4 (Spark 2) to 9.1 (Spark 3)

Atanu — Sat, 12 Feb 2022 16:30:35 GMT

@Jeffrey Mak the doc is not updated yet , may be team is working on it . you can consider 7.x+ for 9 DBR. Please let us know how that goes. Thanks.