Hive version after Upgrade Azure Databricks from 6.4 (Spark 2) to 9.1 (Spark 3)

[jeffreym9]

I have upgraded the Azure Databricks from 6.4 to 9.1 which enable me to use Spark3. As far as I know, the Hive version has to be upgraded to 2.3.7 as well as discussed in:

https://community.databricks.com/s/question/0D53f00001HKHy2CAH/how-to-upgrade-internal-hive-metadata...

I have tried not given those options and continue using Hive 0.13 to run my Spark3 application and everything seems fine. Is it a must to upgrade the Hive metastore to 2.3.7 for Spark3 program to run? Cause everything seems find without adding:

spark.sql.hive.metastore.version 2.3.7
spark.sql.hive.metastore.jars builtin

[Kaniz]

Hi @ jeffreym9! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will get back to you soon. Thanks.

[Atanu]

@Jeffrey Mak​  all supported version is mention here https://docs.microsoft.com/en-us/azure/databricks/data/metastores/external-hive-metastore also, I do not think builtin will work.

• For all other Hive versions, Azure Databricks recommends that you download the metastore JARs and set the configuration 
• spark.sql.hive.metastore.jars
•  
•  to point to the downloaded JARs using the procedure described in Download the metastore jars and point to them.

[Anonymous]

@Jeffrey Mak​ - Does Atanu's answer resolve the issue for you? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?

[jeffreym9]

I'm asking about Datatricks version 9.1. I've follow the url given (https://docs.microsoft.com/en-us/azure/databricks/data/metastores/external-hive-metastore). Do you mind letting me know where in the table is mentioning the supported hive version for Databricks 9.1?

[Atanu · Accepted solution]

@Jeffrey Mak​  the doc is not updated yet , may be team is working on it . you can consider 7.x+ for 9 DBR. Please let us know how that goes. Thanks.