cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

How to upgrade internal hive metadata store version

AlexDavies
Contributor

Is it possible to upgrade the out of the box hive metastore version?

running spark.conf.get("spark.sql.hive.metastore.version") indicates that it is running on 0.13.0

However https://docs.microsoft.com/en-us/azure/databricks/release-notes/runtime/7.x-migration#apache-hive seems to indicate that the version was upgraded to 2.3

I have attempted to add the following to the spark config

spark.sql.hive.metastore.version 2.3.7
spark.sql.hive.metastore.jars builtin

But it results in errors whose stacktrace seems to indicate its trying to connect to an external metastore. Im not interested in setting up an external metastore at this time

What should the hive metastore version be and is there anything I need to do to upgrade it?

1 ACCEPTED SOLUTION

Accepted Solutions

Atanu
Esteemed Contributor
Esteemed Contributor
  1. spark.sql.hive.metastore.jars builtin

can you try changing the same as per

Hive 2.3.7 (Databricks Runtime 7.0 and above): set spark.sql.hive.metastore.jars to builtin.

For all other Hive versions, Azure Databricks recommends that you download the metastore JARs and set the configuration spark.sql.hive.metastore.jars to point to the downloaded JARs using the procedure described in Download the metastore jars and point to them.

https://docs.microsoft.com/en-us/azure/databricks/data/metastores/external-hive-metastore#--spark-co...

View solution in original post

9 REPLIES 9

prasadvaze
Valued Contributor II

Only way I found to solve this is to use external metastore so I am using azure sql db hosted hive metastore . It creates dbo.version table with a row showing version of metastore. Update this row to match the value with spark.sql.hive.metastore.version value

jeffreym9
New Contributor III

I have exactly the same issue. My question is, is it a must to use 2.3.7? or it is still ok to use 0.13.0 in Spark3?

Kaniz
Community Manager
Community Manager

jeffreym9
New Contributor III

That does not help. The suggestion to by-pass the version validation only delays the issue. When i tried to create new tables, it complains that some columns is found missing in the hive metastore (0.13) .

Atanu
Esteemed Contributor
Esteemed Contributor
  1. spark.sql.hive.metastore.jars builtin

can you try changing the same as per

Hive 2.3.7 (Databricks Runtime 7.0 and above): set spark.sql.hive.metastore.jars to builtin.

For all other Hive versions, Azure Databricks recommends that you download the metastore JARs and set the configuration spark.sql.hive.metastore.jars to point to the downloaded JARs using the procedure described in Download the metastore jars and point to them.

https://docs.microsoft.com/en-us/azure/databricks/data/metastores/external-hive-metastore#--spark-co...

That doesn't seem to solve the problem.

We appear to be using hive 0.13.0, docs mention we should be be on 2.3.7. Is there something we have to do on our end to upgrade?

Running the queries gives

spark.conf.get("spark.sql.hive.metastore.jars") //builtin
spark.conf.get("spark.sql.hive.metastore.version") //0.13.0

setting spark.sql.hive.metastore.jars to builtin does not change the metastore version

How do we upgrade our builtin metastore?

Kaniz
Community Manager
Community Manager

Hi @Alex Davies​ ,

  • You can try to change the settings in "spark.sql.hive.metastore.version" in the Azure Databricks cluster configuration and restart.
spark.sql.hive.metastore.version 2.3
spark.sql.hive.metastore.jars builtin

Anonymous
Not applicable

@Alex Davies​ - Does Atanu's information help resolve the issue? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?

pantelis_mare
Contributor III

Hello guys!

Atanu's post, although correct does not solve the problem. Is there any official documentation on how to upgrade the internal databricks metastore to a greater version? If this is availble then we can try Atanu's solution (not sure if needed in that case)

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!