Utility : Remorph (Databricks)
Issue: 'Catalog hive_metastore doesn't exist. Create it?'
Error When Installing Reconcile
I am encountering an issue while installing Reconcile on Databricks. Despite hive_metastore catalog is by default present in the Databricks, I received the following error:
"Catalog `hive_metastore` doesn't exist. Create it? (default: no):"
Here are the steps I have taken to troubleshoot and resolve the issue, but the error persists:
1. Cluster Configuration:
Added the following Spark configuration while creating the cluster:
spark.sql.catalogImplementation=hive
2. Environment Variable:
Set the default catalog using the command:
export SPARK_SQL_CATALOG=hive_metastore
3. Verification:
Ran the SQL command SHOW CATALOGS; in a notebook, and it lists 'hive_metastore' as an available catalog.
4. Databricks Runtime:
The cluster is running Databricks Runtime 15.4 LTS, which should support the 'hive_metastore' catalog by default.
Despite these configurations, the error persists. It appears that Reconcile does not recognize the hive_metastore catalog, even though it is accessible and functional through the Databricks environment.
Requesting for a help on resolving the same, if possible, steps to resolve.
#remorph #reconcile #installation error
