Is it possible to set up external hive metastore with DBSQL?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-24-2021 09:49 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-25-2021 02:31 PM
Yes, it's possible to connect to an external metastore. The configurations remain the same as a normal Databricks cluster.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-09-2021 07:41 AM
@Harikrishnan Kunhumveettil Using Azure databricks, I have set up SQL Endpoint with the connection details that match with global init script. I am able to browse tables from regular cluster in Data Engineering module but i get below error when trying a query using SQL Endpoint
*****************
on the schema browser left pane error : - Failed to load Databases list.
on the query results pane error:-
Error running query
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
************************
The SQL end point settings are as below - (user id and passwd are in azure key vault)
************************
spark.sql.hive.metastore.* true
spark.hadoop.fs.azure.account.auth.type.<storage_acct>.dfs.core.windows.net OAuth
spark.hadoop.fs.azure.account.oauth.provider.type.<storage_acct>.dfs.core.windows.net org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider
spark.hadoop.fs.azure.account.oauth2.client.id.<storage_acct>.dfs.core.windows.net {{secrets/<secret_scope>/client-id}}
spark.hadoop.fs.azure.account.oauth2.client.secret.<storage_acct>.dfs.core.windows.net {{secrets/<secret_scope>/client-secret}}
spark.hadoop.fs.azure.account.oauth2.client.endpoint.<storage_acct>.dfs.core.windows.net https://login.microsoftonline.com/<tenant_id>/oauth2/token
spark.hadoop.javax.jdo.option.ConnectionURL jdbc://<sql server name>.database.windows.net:1433;database=<db_name>
spark.hadoop.javax.jdo.option.ConnectionUserName metastoredbuser
spark.hadoop.javax.jdo.option.ConnectionPassword {{secrets/<secret_scope>/metastoredbpwd}}
spark.hadoop.javax.jdo.option.ConnectionDriverName com.microsoft.sqlserver.jdbc.SQLServerDriver
spark.sql.hive.metastore.version 2.3.7
spark.sql.hive.metastore.jars /databricks/hive_metastore_jars/*
************************