brickster_2018
Databricks Employee
Databricks Employee

Databricks by default uses hive metastore version 0.13. However, it's possible to switch to a newer client version connecting to the internal Databricks metastore.

Details are available here:

https://docs.databricks.com/data/metastores/external-hive-metastore.html#spark-configuration-options

Set spark.sql.hive.metastore.version to the version of your Hive metastore and spark.sql.hive.metastore.jars as follows:
 
Hive 0.13: do not set spark.sql.hive.metastore.jars.
 
Hive 1.2.0 or 1.2.1 (Databricks Runtime 6.6 and below): set spark.sql.hive.metastore.jars to builtin.

When it comes to external metastore, based on the use case and need a version can be selected. We recommend checking the compatibility matrix before deciding on the version.

https://docs.databricks.com/data/metastores/external-hive-metastore.html#external-apache-hive-metast...

View solution in original post