- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-23-2021 03:59 PM
I am setting up an external metastore to connect my Databricks cluster. Which is the preferred and recommended Hive metastore version? Also are there any preference or recommendations on the database instance size/type
- Labels:
-
Hive metastore
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-24-2021 06:28 AM
Databricks by default uses hive metastore version 0.13. However, it's possible to switch to a newer client version connecting to the internal Databricks metastore.
Details are available here:
https://docs.databricks.com/data/metastores/external-hive-metastore.html#spark-configuration-options
Set spark.sql.hive.metastore.version to the version of your Hive metastore and spark.sql.hive.metastore.jars as follows:
Hive 0.13: do not set spark.sql.hive.metastore.jars.
Hive 1.2.0 or 1.2.1 (Databricks Runtime 6.6 and below): set spark.sql.hive.metastore.jars to builtin.When it comes to external metastore, based on the use case and need a version can be selected. We recommend checking the compatibility matrix before deciding on the version.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-22-2021 09:44 PM
@Harikrishnan Kunhumveettil we use databricks runtime 7.3LTS and 9.1LTS. And external hive metastore hosted on azue sql db. Using global init script I have set spark.sql.hive.metastore.version 2.3.7 and downloaded spark.sql.hive.metastore.jars from maven.
To create tables in metastore sql server db (not the delta tables but the metastore tables) , I used a sql script from apache hive website. There is dbo.version table with one row which defines the metastore version. How do I upgrade to metastore 3.1.0 version?
