cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

External metastore version

brickster_2018
Esteemed Contributor

I am setting up an external metastore to connect my Databricks cluster. Which is the preferred and recommended Hive metastore version? Also are there any preference or recommendations on the database instance size/type

1 ACCEPTED SOLUTION

Accepted Solutions

brickster_2018
Esteemed Contributor

Databricks by default uses hive metastore version 0.13. However, it's possible to switch to a newer client version connecting to the internal Databricks metastore.

Details are available here:

https://docs.databricks.com/data/metastores/external-hive-metastore.html#spark-configuration-options

Set spark.sql.hive.metastore.version to the version of your Hive metastore and spark.sql.hive.metastore.jars as follows:
 
Hive 0.13: do not set spark.sql.hive.metastore.jars.
 
Hive 1.2.0 or 1.2.1 (Databricks Runtime 6.6 and below): set spark.sql.hive.metastore.jars to builtin.

When it comes to external metastore, based on the use case and need a version can be selected. We recommend checking the compatibility matrix before deciding on the version.

https://docs.databricks.com/data/metastores/external-hive-metastore.html#external-apache-hive-metast...

View solution in original post

2 REPLIES 2

brickster_2018
Esteemed Contributor

Databricks by default uses hive metastore version 0.13. However, it's possible to switch to a newer client version connecting to the internal Databricks metastore.

Details are available here:

https://docs.databricks.com/data/metastores/external-hive-metastore.html#spark-configuration-options

Set spark.sql.hive.metastore.version to the version of your Hive metastore and spark.sql.hive.metastore.jars as follows:
 
Hive 0.13: do not set spark.sql.hive.metastore.jars.
 
Hive 1.2.0 or 1.2.1 (Databricks Runtime 6.6 and below): set spark.sql.hive.metastore.jars to builtin.

When it comes to external metastore, based on the use case and need a version can be selected. We recommend checking the compatibility matrix before deciding on the version.

https://docs.databricks.com/data/metastores/external-hive-metastore.html#external-apache-hive-metast...

prasadvaze
Valued Contributor II

@Harikrishnan Kunhumveettilโ€‹  we use databricks runtime 7.3LTS and 9.1LTS. And external hive metastore hosted on azue sql db. Using global init script I have set spark.sql.hive.metastore.version 2.3.7 and downloaded spark.sql.hive.metastore.jars from maven.

To create tables in metastore sql server db (not the delta tables but the metastore tables) , I used a sql script from apache hive website. There is dbo.version table with one row which defines the metastore version. How do I upgrade to metastore 3.1.0 version?

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group