- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-09-2022 08:41 PM
We are using Dataricks in AWS. i am not able to run a notebook even when cluster is running. When i run a cell, it returns "cancel". When i check the event log for the cluster, it shows "Metastore is down". Couldn't see any databases or tables that i created in "data" tab, Its keeps on showing loading and returns some error there.
Appreciate a help on this.
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-13-2022 05:34 AM
This means the network is fine, but something in the spark config is amiss.
What are the DBR version and the hive version? Please check f you are using a compatible version.
If you don't specify any version, it will take 1.3 and you wouldn't have to use jars in this case.
Please use below config with your appropriate values,
spark.hadoop.datanucleus.fixedDatastore false
spark.hadoop.datanucleus.autoCreateSchema true
spark.hadoop.datanucleus.autoCreateTables true
spark.hadoop.javax.jdo.option.ConnectionURL jdbc:mariadb://<database-name>.cj11tymkwz5w.us-west-2.rds.amazonaws.com
:3306/databricks_metastore?createDatabaseIfNotExist=true
spark.hadoop.javax.jdo.option.ConnectionDriverName org.mariadb.jdbc.Driver
spark.hadoop.javax.jdo.option.ConnectionUserName <value>
spark.hadoop.javax.jdo.option.ConnectionPassword <value>
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-09-2022 11:39 PM
do you happen to use an external metastore?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-10-2022 01:34 AM
@Werner Stinckens No we dont use external metastore
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-10-2022 01:40 AM
ok that is not normal behaviour.
Is it a new databricks workspace, community or premium or standard?
seems like something is wrong with the workspace deployment, because databricks uses a default metastore which works stable.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-10-2022 02:07 AM
Yes, i have created new Databricks Trial account (on AWS) (Premium)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-10-2022 02:10 AM
@Werner Stinckens I have created workspace using "custom AWS configuration". It was working fine till yesterday but suddenly returns this error.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-10-2022 02:15 AM
Seems like a workspace deployment issue.
But I see you have another topic open, which seems related tbh.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-10-2022 05:32 AM
Metastore is hosted in AWS RDS.
Additionally you can check cluster start logs. In logs is metastore url and connection message, probably you just blocked traffic to that.
On that page you can also find urls/ips which should be allowed out/routed through internet gateway: https://docs.databricks.com/administration-guide/cloud-configurations/aws/customer-managed-vpc.html
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-11-2022 12:57 AM
As suggested by @Hubert Dudek , please check your network settings, in case you have any firewall or rules that may be blocking the RDS connection on port 3306.
Also, check if you are whitelisting the correct endpoint as per your region.
To debug this further, you can SSH into one of the AWS worker instances used by a Databricks cluster and test the connectivity.
Eg: For us-east-1 region, run this command
nc -zv mdb7sywh50xhpr.chkweekm4xjq.us-east-1.rds.amazonaws.com 3306
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-13-2022 10:14 PM
@Hubert Dudek @Kavya Manohar Parag
Thanks for the input. I have tried running below command, it was succeeded.
"!nc -zv md1n4trqmokgnhr.csnrqwqko4ho.ap-southeast-1.rds.amazonaws.com 3306".
However, Database is not accessible. Please see the error comes for show databases.
Also, I have tried several trial accounts using this AWS account. Is there any restriction for trial accounts ?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-17-2022 10:46 AM
So this means still you need to check the config of your External metastore.
If you run below - whats the output -
%scala
val metastoreURL = spark.sparkContext.hadoopConfiguration.get(“javax.jdo.option.ConnectionURL”)
val metastoreUser = spark.sparkContext.hadoopConfiguration.get(“javax.jdo.option.ConnectionUserName”)
val metastorePassword = spark.sparkContext.hadoopConfiguration.get(“javax.jdo.option.ConnectionPassword”) print(metastoreURL)
print(metastoreUser)
also, it's worth try to recheck the configuration .
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-13-2022 05:34 AM
This means the network is fine, but something in the spark config is amiss.
What are the DBR version and the hive version? Please check f you are using a compatible version.
If you don't specify any version, it will take 1.3 and you wouldn't have to use jars in this case.
Please use below config with your appropriate values,
spark.hadoop.datanucleus.fixedDatastore false
spark.hadoop.datanucleus.autoCreateSchema true
spark.hadoop.datanucleus.autoCreateTables true
spark.hadoop.javax.jdo.option.ConnectionURL jdbc:mariadb://<database-name>.cj11tymkwz5w.us-west-2.rds.amazonaws.com
:3306/databricks_metastore?createDatabaseIfNotExist=true
spark.hadoop.javax.jdo.option.ConnectionDriverName org.mariadb.jdbc.Driver
spark.hadoop.javax.jdo.option.ConnectionUserName <value>
spark.hadoop.javax.jdo.option.ConnectionPassword <value>
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-25-2022 02:04 PM
Hi @Dhusanth Thangavadivel ,
Just a friendly follow-up. Did @Kavya Manohar Parag 's response help you to resolved this issue?