Databricks

alesventus · ‎07-25-2023

When I want to run notebook in databricks all queries, saves and read take really long and I found error message in the clusters event log that says: Metastore is down. So, I think cluster is not able to connect to the metastore right now. Could be this the reason? I tried to create another cluster but with the same result. Any help on this? Unity catalog`s Metastore is on ADLS gen2. and we use premium workspace.

After few minutes the command is finished. But simple load of delta table with few records takes 5 minutes or so.

Thanks

Kaniz · ‎07-26-2023

Hi @alesventus, Yes, the cluster may becannot connect to the metastore, which could be the reason for the slow performance and error messages. You can check the eventNodeDiskSpace metric in usage_logs to see if the root file system of the driver's host VM is getting 100%, which can cause the cluster to go down.

Additionally, it is recommended to convert the table to a delta table to improve performance in Unity Catalog. You can also try increasing the number of user connections from the RDS and performing instance-type upgrades.

If the issue persists, you can check the node daemon log of the problematic instance and raise the case to the Dataplane team.

alesventus · ‎07-26-2023

Hi Kaniz. Thanks for reply. Just want to add some information. Im not sure if this error message is related to legacy Hive metastore or to Unity Catalog. We use UC from the beginning. Our metastore is on external storage - see screenshot.

This error occurs in the clusters event log exactly 6 minutes after the start of the cluster. Every time we start the cluster that error is there.

Angela09 · ‎07-27-2023

You can verify the status of the Metastore in your Azure Data Lake Storage Gen2 (ADLS Gen2) and ensure that it is up and running. If there is an issue with the Metastore service, it can impact the performance of queries and operations in Databricks.

myfedloan