07-25-2023 10:31 AM - edited 07-25-2023 11:11 AM
When I want to run notebook in databricks all queries, saves and read take really long and I found error message in the clusters event log that says: Metastore is down. So, I think cluster is not able to connect to the metastore right now. Could be this the reason? I tried to create another cluster but with the same result. Any help on this? Unity catalog`s Metastore is on ADLS gen2. and we use premium workspace.
After few minutes the command is finished. But simple load of delta table with few records takes 5 minutes or so.
Thanks
07-26-2023 02:34 AM - edited 07-26-2023 02:35 AM
Hi @alesventus, Yes, the cluster may becannot connect to the metastore, which could be the reason for the slow performance and error messages. You can check the eventNodeDiskSpace metric in usage_logs to see if the root file system of the driver's host VM is getting 100%, which can cause the cluster to go down.
Additionally, it is recommended to convert the table to a delta table to improve performance in Unity Catalog. You can also try increasing the number of user connections from the RDS and performing instance-type upgrades.
If the issue persists, you can check the node daemon log of the problematic instance and raise the case to the Dataplane team.
07-26-2023 11:34 PM
Hi Kaniz. Thanks for reply. Just want to add some information. Im not sure if this error message is related to legacy Hive metastore or to Unity Catalog. We use UC from the beginning. Our metastore is on external storage - see screenshot.
This error occurs in the clusters event log exactly 6 minutes after the start of the cluster. Every time we start the cluster that error is there.
07-28-2023 09:43 AM
@alesventus what is size of your data and what type of cluster size you are using. also is your workspace region and metastore region same
07-02-2024 08:42 AM
@alesventus I think "Metastore is down" here is related to the legacy Hive metastore. You can find the stack trace in the driver logs.
In your case do you need Hivemetastore?
07-04-2024 12:04 AM
This issue is solely related to the VNET. Azure engineer must set up connection within VNET correctly.
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group