External metastore version

[brickster_2018]

I am setting up an external metastore to connect my Databricks cluster. Which is the preferred and recommended Hive metastore version? Also are there any preference or recommendations on the database instance size/type

[brickster_2018 · Accepted solution]

Databricks by default uses hive metastore version 0.13. However, it's possible to switch to a newer client version connecting to the internal Databricks metastore.

Details are available here:

https://docs.databricks.com/data/metastores/external-hive-metastore.html#spark-configuration-options

Set spark.sql.hive.metastore.version to the version of your Hive metastore and spark.sql.hive.metastore.jars as follows:
 
Hive 0.13: do not set spark.sql.hive.metastore.jars.
 
Hive 1.2.0 or 1.2.1 (Databricks Runtime 6.6 and below): set spark.sql.hive.metastore.jars to builtin.

When it comes to external metastore, based on the use case and need a version can be selected. We recommend checking the compatibility matrix before deciding on the version.

https://docs.databricks.com/data/metastores/external-hive-metastore.html#external-apache-hive-metast...

[prasadvaze]

@Harikrishnan Kunhumveettil​  we use databricks runtime 7.3LTS and 9.1LTS. And external hive metastore hosted on azue sql db. Using global init script I have set spark.sql.hive.metastore.version 2.3.7 and downloaded spark.sql.hive.metastore.jars from maven.

To create tables in metastore sql server db (not the delta tables but the metastore tables) , I used a sql script from apache hive website. There is dbo.version table with one row which defines the metastore version. How do I upgrade to metastore 3.1.0 version?