Utility : Remorph (Databricks)

Issue: 'Catalog  hive_metastore doesn't exist. Create it?'

Error When Installing Reconcile
I am encountering an issue while installing Reconcile on Databricks. Despite hive_metastore catalog is by default present in the Databricks, I received the following error:

"Catalog `hive_metastore` doesn't exist. Create it? (default: no):"

Here are the steps I have taken to troubleshoot and resolve the issue, but the error persists:

1. Cluster Configuration:
     Added the following Spark configuration while creating the cluster:
     spark.sql.catalogImplementation=hive 

2. Environment Variable:
     Set the default catalog using the command:
     export SPARK_SQL_CATALOG=hive_metastore

3. Verification:
     Ran the SQL command SHOW CATALOGS; in a notebook, and it lists 'hive_metastore' as an available catalog.

4. Databricks Runtime:
      The cluster is running Databricks Runtime 15.4 LTS, which should support the 'hive_metastore' catalog by default.

Despite these configurations, the error persists. It appears that Reconcile does not recognize the hive_metastore catalog, even though it is accessible and functional through the Databricks environment.

Requesting for a help on resolving the same, if possible, steps to resolve. 

 #remorph #reconcile #installation error