How to upgrade internal hive metadata store version

AlexDavies
Contributor

Is it possible to upgrade the out of the box hive metastore version?

running spark.conf.get("spark.sql.hive.metastore.version") indicates that it is running on 0.13.0

However https://docs.microsoft.com/en-us/azure/databricks/release-notes/runtime/7.x-migration#apache-hive seems to indicate that the version was upgraded to 2.3

I have attempted to add the following to the spark config

spark.sql.hive.metastore.version 2.3.7
spark.sql.hive.metastore.jars builtin

But it results in errors whose stacktrace seems to indicate its trying to connect to an external metastore. Im not interested in setting up an external metastore at this time

What should the hive metastore version be and is there anything I need to do to upgrade it?