topic Not able to run notebook even when cluster is running and databases/tables are not visible in "data" tab. in Data Engineering

Not able to run notebook even when cluster is running and databases/tables are not visible in "data" tab.

Anonymous — Thu, 10 Mar 2022 04:41:51 GMT

We are using Dataricks in AWS. i am not able to run a notebook even when cluster is running. When i run a cell, it returns "cancel". When i check the event log for the cluster, it shows "Metastore is down". Couldn't see any databases or tables that i created in "data" tab, Its keeps on showing loading and returns some error there.

Appreciate a help on this.

Re: Not able to run notebook even when cluster is running and databases/tables are not visible in "data" tab.

-werners- — Thu, 10 Mar 2022 07:39:01 GMT

do you happen to use an external metastore?

Re: Not able to run notebook even when cluster is running and databases/tables are not visible in "data" tab.

Anonymous — Thu, 10 Mar 2022 09:34:44 GMT

@Werner Stinckens No we dont use external metastore

Re: Not able to run notebook even when cluster is running and databases/tables are not visible in "data" tab.

-werners- — Thu, 10 Mar 2022 09:40:54 GMT

ok that is not normal behaviour.

Is it a new databricks workspace, community or premium or standard?

seems like something is wrong with the workspace deployment, because databricks uses a default metastore which works stable.

Re: Not able to run notebook even when cluster is running and databases/tables are not visible in "data" tab.

Anonymous — Thu, 10 Mar 2022 10:07:37 GMT

Yes, i have created new Databricks Trial account (on AWS) (Premium)

Re: Not able to run notebook even when cluster is running and databases/tables are not visible in "data" tab.

Anonymous — Thu, 10 Mar 2022 10:10:50 GMT

@Werner Stinckens I have created workspace using "custom AWS configuration". It was working fine till yesterday but suddenly returns this error.

Re: Not able to run notebook even when cluster is running and databases/tables are not visible in "data" tab.

-werners- — Thu, 10 Mar 2022 10:15:17 GMT

Seems like a workspace deployment issue.

But I see you have another topic open, which seems related tbh.

Re: Not able to run notebook even when cluster is running and databases/tables are not visible in "data" tab.

Hubert-Dudek — Thu, 10 Mar 2022 13:32:14 GMT

Metastore is hosted in AWS RDS.

Additionally you can check cluster start logs. In logs is metastore url and connection message, probably you just blocked traffic to that.

On that page you can also find urls/ips which should be allowed out/routed through internet gateway: https://docs.databricks.com/administration-guide/cloud-configurations/aws/customer-managed-vpc.html

Re: Not able to run notebook even when cluster is running and databases/tables are not visible in "data" tab.

User16753725182 — Fri, 11 Mar 2022 08:57:24 GMT

As suggested by @Hubert Dudek , please check your network settings, in case you have any firewall or rules that may be blocking the RDS connection on port 3306.

Also, check if you are whitelisting the correct endpoint as per your region.

To debug this further, you can SSH into one of the AWS worker instances used by a Databricks cluster and test the connectivity.

Eg: For us-east-1 region, run this command

nc -zv mdb7sywh50xhpr.chkweekm4xjq.us-east-1.rds.amazonaws.com 3306

Re: Not able to run notebook even when cluster is running and databases/tables are not visible in "data" tab.

Anonymous — Mon, 14 Mar 2022 05:14:31 GMT

@Hubert Dudek @Kavya Manohar Parag

Thanks for the input. I have tried running below command, it was succeeded.

"!nc -zv md1n4trqmokgnhr.csnrqwqko4ho.ap-southeast-1.rds.amazonaws.com 3306".

However, Database is not accessible. Please see the error comes for show databases.

Also, I have tried several trial accounts using this AWS account. Is there any restriction for trial accounts ?

Re: Not able to run notebook even when cluster is running and databases/tables are not visible in "data" tab.

User16753725182 — Wed, 13 Apr 2022 12:34:41 GMT

This means the network is fine, but something in the spark config is amiss.

What are the DBR version and the hive version? Please check f you are using a compatible version.

If you don't specify any version, it will take 1.3 and you wouldn't have to use jars in this case.

Please use below config with your appropriate values,

spark.hadoop.datanucleus.fixedDatastore false
 
spark.hadoop.datanucleus.autoCreateSchema true
 
spark.hadoop.datanucleus.autoCreateTables true
 
spark.hadoop.javax.jdo.option.ConnectionURL jdbc:mariadb://<database-name>.cj11tymkwz5w.us-west-2.rds.amazonaws.com
 
:3306/databricks_metastore?createDatabaseIfNotExist=true
 
spark.hadoop.javax.jdo.option.ConnectionDriverName org.mariadb.jdbc.Driver
 
spark.hadoop.javax.jdo.option.ConnectionUserName <value> 
 
spark.hadoop.javax.jdo.option.ConnectionPassword <value>

Re: Not able to run notebook even when cluster is running and databases/tables are not visible in "data" tab.

Atanu — Sun, 17 Apr 2022 17:46:00 GMT

So this means still you need to check the config of your External metastore.

If you run below - whats the output -

%scala

val metastoreURL = spark.sparkContext.hadoopConfiguration.get(“javax.jdo.option.ConnectionURL”)

val metastoreUser = spark.sparkContext.hadoopConfiguration.get(“javax.jdo.option.ConnectionUserName”)

val metastorePassword = spark.sparkContext.hadoopConfiguration.get(“javax.jdo.option.ConnectionPassword”) print(metastoreURL)

print(metastoreUser)

also, it's worth try to recheck the configuration .

Re: Not able to run notebook even when cluster is running and databases/tables are not visible in "data" tab.

jose_gonzalez — Mon, 25 Apr 2022 21:04:23 GMT

Hi @Dhusanth Thangavadivel ,

Just a friendly follow-up. Did @Kavya Manohar Parag 's response help you to resolved this issue?