cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

External metastore version

brickster_2018
Esteemed Contributor
Esteemed Contributor

I am setting up an external metastore to connect my Databricks cluster. Which is the preferred and recommended Hive metastore version? Also are there any preference or recommendations on the database instance size/type

1 ACCEPTED SOLUTION

Accepted Solutions

brickster_2018
Esteemed Contributor
Esteemed Contributor

Databricks by default uses hive metastore version 0.13. However, it's possible to switch to a newer client version connecting to the internal Databricks metastore.

Details are available here:

https://docs.databricks.com/data/metastores/external-hive-metastore.html#spark-configuration-options

Set spark.sql.hive.metastore.version to the version of your Hive metastore and spark.sql.hive.metastore.jars as follows:
 
Hive 0.13: do not set spark.sql.hive.metastore.jars.
 
Hive 1.2.0 or 1.2.1 (Databricks Runtime 6.6 and below): set spark.sql.hive.metastore.jars to builtin.

When it comes to external metastore, based on the use case and need a version can be selected. We recommend checking the compatibility matrix before deciding on the version.

https://docs.databricks.com/data/metastores/external-hive-metastore.html#external-apache-hive-metast...

View solution in original post

2 REPLIES 2

brickster_2018
Esteemed Contributor
Esteemed Contributor

Databricks by default uses hive metastore version 0.13. However, it's possible to switch to a newer client version connecting to the internal Databricks metastore.

Details are available here:

https://docs.databricks.com/data/metastores/external-hive-metastore.html#spark-configuration-options

Set spark.sql.hive.metastore.version to the version of your Hive metastore and spark.sql.hive.metastore.jars as follows:
 
Hive 0.13: do not set spark.sql.hive.metastore.jars.
 
Hive 1.2.0 or 1.2.1 (Databricks Runtime 6.6 and below): set spark.sql.hive.metastore.jars to builtin.

When it comes to external metastore, based on the use case and need a version can be selected. We recommend checking the compatibility matrix before deciding on the version.

https://docs.databricks.com/data/metastores/external-hive-metastore.html#external-apache-hive-metast...

prasadvaze
Valued Contributor II

@Harikrishnan Kunhumveettil​  we use databricks runtime 7.3LTS and 9.1LTS. And external hive metastore hosted on azue sql db. Using global init script I have set spark.sql.hive.metastore.version 2.3.7 and downloaded spark.sql.hive.metastore.jars from maven.

To create tables in metastore sql server db (not the delta tables but the metastore tables) , I used a sql script from apache hive website. There is dbo.version table with one row which defines the metastore version. How do I upgrade to metastore 3.1.0 version?

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!