topic Re: Databricks SQL not displaying all the databases that i have on my cluster. in Data Engineering

Databricks SQL not displaying all the databases that i have on my cluster.

DB_007 — Mon, 15 Nov 2021 16:37:34 GMT

I have a cluster running on 7.3 LTS and it has about 35+ databases. When i tried to setup an endpoint on Databricks SQL, i do not see any database listed.

Re: Databricks SQL not displaying all the databases that i have on my cluster.

Atanu — Mon, 15 Nov 2021 16:45:38 GMT

Could you please share a screenshot?

Re: Databricks SQL not displaying all the databases that i have on my cluster.

DB_007 — Mon, 15 Nov 2021 20:38:11 GMT

Attached pic

Re: Databricks SQL not displaying all the databases that i have on my cluster.

Atanu — Mon, 15 Nov 2021 21:20:44 GMT

@Arif Ali Not seeing the pic. Could you please upload again.

Re: Databricks SQL not displaying all the databases that i have on my cluster.

DB_007 — Mon, 15 Nov 2021 21:46:30 GMT

Re: Databricks SQL not displaying all the databases that i have on my cluster.

Atanu — Mon, 15 Nov 2021 22:38:58 GMT

Another question @Arif Ali what is the permission level you inherit from this cluster?

Re: Databricks SQL not displaying all the databases that i have on my cluster.

Anonymous — Mon, 15 Nov 2021 22:44:29 GMT

@Arif Ali There might be a couple of reasons.

Table ACL is enabled by default in DB SQL, so please ensure that you have permission to the underlying databases as mentioned in https://docs.databricks.com/sql/user/security/data-access-overview.html

If you're using an external metastore, please add the required Data access configurations as mentioned in https://docs.databricks.com/sql/admin/data-access-configuration.html

Re: Databricks SQL not displaying all the databases that i have on my cluster.

DB_007 — Tue, 16 Nov 2021 00:13:23 GMT

I have admin privileges on the cluster and also full access to the endpoint. Metastore is external and do i need to do anything in order to get databases listed on the Databricks SQL.

Re: Databricks SQL not displaying all the databases that i have on my cluster.

User16871418122 — Wed, 17 Nov 2021 03:04:02 GMT

hi @Arif Ali You may have to check the data access config to add the params for external metastore:

spark.hadoop.javax.jdo.option.ConnectionDriverName org.mariadb.jdbc.Driver

spark.hadoop.javax.jdo.option.ConnectionUserName <mysql-username>

spark.hadoop.javax.jdo.option.ConnectionPassword <mysql-password>

spark.sql.hive.metastore.version <hive-version>

spark.sql.hive.metastore.jars <hive-jar-source>

If the data access config is already added, it is likely the metastore initialization is taking longer. ie( spark to apply the spark configs to load metastore). If given some more time it will initialize the metastore and enables schema browsing. If still issues, the configs may be wrong, you shall have them validated on interactive Databricks cluster before applying in SQL endpoints.

In the near future with the Unity catalog, the schema browsing is cluster less. The dependency on having a cluster is removed. 😊