Unit hive-metastore.service not found

[Surya0]

Hi Everyone,

I've encountered an issue while trying to make use of the hive-metastore capability in Databricks to create a new database and table for our latest use case. The specific command I used was "create database if not exists newDB". However, I've noticed that the cluster operation has been running unusually long (over 10 minutes), which is not typical based on my experience.

In an attempt to troubleshoot, I came across a suggestion to restart the hive-metastore service using the command "%sh sudo service hive-metastore restart". This, however, resulted in an error: "Unit hive-metastore.service not found".

I've tried restarting the existing clusters and even created a new Standard_DS3_V2 cluster with 13.3 LTS runtime, but the issue persists. I'm wondering if I'm missing something in my approach or if there's a potential configuration issue. Could anyone advise on further steps I could take to diagnose this problem? Are there specific logs or settings I should be checking?

Any insights or suggestions would be greatly appreciated.

Thank you!

[Kaniz]

Hello Surya0! It seems you’re facing a hiccup with the hive-metastore service in Databricks.

Let’s dive into some troubleshooting steps to get this sorted out:

Check Metastore Configuration:

• Ensure you have no custom Hive metastore configurations on your cluster. Sometimes, these settings can cause unexpected behaviour.
• If you’re using an external metastore, consider increasing its connection limit, especially if you h....

Access Denied Error:

• The error message “Access denied for user ‘hive’@‘sandbox.hortonworks.com’” suggests a MySQL authentication issue. Ensure the user ‘hive’ has the necessary privileges in your MySQL database.
• Verify the user’s grants using the command: show grants for 'hive'@'sandbox.hortonworks.com';

Schema Initialization:

• If the hive-metastore schema is not available, you might encounter issues. Ensure that the following flag is enabled in your init scripts to create the hive Schema and tables ....

Hive Metastore Location:

• Double-check the location of your Hive Metastore. Sometimes, pointing Hive to the correct store can resolve issues.
• Initialize the database in a location of your choice (e.g., MySQL) and configure Hive to use it.

Logs and Diagnostics:

• Check the logs for any specific error messages. You can find them in the Databricks workspace or the cluster driver logs.
• Look for any exceptions related to the metastore service or database connectivity.

[Surya0]

Thanks for the debugging steps It helped me understand that Azure Databricks cannot reach the MYSQL server. I see this error now. 
"Failed to initialize pool: Could not connect to address=(host=consolidated-northeuropec2-prod-metastore-2.mysql.database.azure.com)(port=3306)(type=master) : Socket fail to connect to host:consolidated-northeuropec2-prod-metastore-2.mysql.database.azure.com, port:3306. connect timed out"  
From the research, it sounded like the problem with the firewall. 
I enabled the inbound connection for port 3306. (I am not quite sure I did correctly or not). Please share your thoughts on this error.
Thanks

[Surya0 · Accepted solution]

Quick update, I resolved the issue. the problem was I created Azure Databricks inside Vnet without opening up default ports for Databricks to connect with MySql. Once I created the route table with the appropriate ports opened I can connect to Metastore. Thanks for the support Kaniz.

[rakeshprasad1]

@Surya0 : i am facing same issue. stack trace is 

Could not connect to address=(host=consolidated-northeuropec2-prod-metastore-2.mysql.database.azure.com)(port=3306)(type=master) : Socket fail to connect to host:consolidated-northeuropec2-prod-metastore-2.mysql.database.azure.com, port:3306. consolidated-northeuropec2-prod-metastore-2.mysql.database.azure.com

My ADB is also running in our VNET.

this is my inbound

and following is my outbound

I did open