topic Re: External metastore version in Data Engineering

External metastore version

brickster_2018 — Wed, 23 Jun 2021 22:59:24 GMT

I am setting up an external metastore to connect my Databricks cluster. Which is the preferred and recommended Hive metastore version? Also are there any preference or recommendations on the database instance size/type

Re: External metastore version

brickster_2018 — Thu, 24 Jun 2021 13:28:55 GMT

Databricks by default uses hive metastore version 0.13. However, it's possible to switch to a newer client version connecting to the internal Databricks metastore.

Details are available here:

https://docs.databricks.com/data/metastores/external-hive-metastore.html#spark-configuration-options

Set spark.sql.hive.metastore.version to the version of your Hive metastore and spark.sql.hive.metastore.jars as follows:
 
Hive 0.13: do not set spark.sql.hive.metastore.jars.
 
Hive 1.2.0 or 1.2.1 (Databricks Runtime 6.6 and below): set spark.sql.hive.metastore.jars to builtin.

When it comes to external metastore, based on the use case and need a version can be selected. We recommend checking the compatibility matrix before deciding on the version.

https://docs.databricks.com/data/metastores/external-hive-metastore.html#external-apache-hive-metastore

Re: External metastore version

prasadvaze — Tue, 23 Nov 2021 05:44:17 GMT

@Harikrishnan Kunhumveettil we use databricks runtime 7.3LTS and 9.1LTS. And external hive metastore hosted on azue sql db. Using global init script I have set spark.sql.hive.metastore.version 2.3.7 and downloaded spark.sql.hive.metastore.jars from maven.

To create tables in metastore sql server db (not the delta tables but the metastore tables) , I used a sql script from apache hive website. There is dbo.version table with one row which defines the metastore version. How do I upgrade to metastore 3.1.0 version?