Hive version after Upgrade Azure Databricks from 6.4 (Spark 2) to 9.1 (Spark 3)

[jeffreym9]

I have upgraded the Azure Databricks from 6.4 to 9.1 which enable me to use Spark3. As far as I know, the Hive version has to be upgraded to 2.3.7 as well as discussed in:

https://community.databricks.com/s/question/0D53f00001HKHy2CAH/how-to-upgrade-internal-hive-metadata...

I have tried not given those options and continue using Hive 0.13 to run my Spark3 application and everything seems fine. Is it a must to upgrade the Hive metastore to 2.3.7 for Spark3 program to run? Cause everything seems find without adding:

spark.sql.hive.metastore.version 2.3.7
spark.sql.hive.metastore.jars builtin

[Atanu]

@Jeffrey Mak​  all supported version is mention here https://docs.microsoft.com/en-us/azure/databricks/data/metastores/external-hive-metastore also, I do not think builtin will work.

• For all other Hive versions, Azure Databricks recommends that you download the metastore JARs and set the configuration 
• spark.sql.hive.metastore.jars
•  
•  to point to the downloaded JARs using the procedure described in Download the metastore jars and point to them.

[Anonymous]

@Jeffrey Mak​ - Does Atanu's answer resolve the issue for you? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?

[jeffreym9]

I'm asking about Datatricks version 9.1. I've follow the url given (https://docs.microsoft.com/en-us/azure/databricks/data/metastores/external-hive-metastore). Do you mind letting me know where in the table is mentioning the supported hive version for Databricks 9.1?

[Atanu · Accepted solution]

@Jeffrey Mak​  the doc is not updated yet , may be team is working on it . you can consider 7.x+ for 9 DBR. Please let us know how that goes. Thanks.