topic Re: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHive in Data Engineering

java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMeta

balwantsingh24 — Fri, 27 Sep 2024 11:13:19 GMT

Guys please help me too solve this issue, I need it very urgent basis.

Re: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHive

saikumar246 — Tue, 01 Oct 2024 07:24:39 GMT

Hi @balwantsingh24 Thank you for sharing your concern here.

Seems like the cluster is unable to initialize the metastore. Are you using the Databricks internal Hive metastore or any external metastore?

We can see this error if the metastore is not set up correctly. I think you can also observe "metastore is down" events in the Cluster event logs during the error.

Make sure the metastore is set up correctly if it is an external metastore on the cluster. you can also check the cluster driver logs (log4j logs) to know the details of the error.

Please leave a like if it is helpful. Follow-ups are appreciated.

Kudos,

Sai Kumar

Re: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHive

balwantsingh24 — Tue, 01 Oct 2024 07:38:24 GMT

It is helpful @saikumar246 , But could you please elaborate the external and internal hive metastore. I need to know how to check cluster driver logs (new here ). What steps to do to initoalise the cluster metastore

Re: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHive

saikumar246 — Tue, 01 Oct 2024 12:37:55 GMT

Hi @balwantsingh24

Internal Metastore:-

Internal metastores are managed by Databricks and are typically used to store metadata about databases, tables, views, and user-defined functions (UDFs). This metadata is essential for operations like the SHOW TABLES SQL command and the Tables UI in Databricks. The internal metastore is part of the Apache Hive project and is hosted by Databricks for their customers.

External Metastore:-

External metastores, on the other hand, are customer-hosted and can be configured to connect to Databricks clusters. This setup is often used to share metadata between Databricks and other systems like Hive, Glue Presto etc; Customers explicitly configure this for their requirements or use-case.

To check the driver logs, you can follow the below link,

https://docs.databricks.com/en/compute/clusters-manage.html#compute-driver-and-worker-logs

Based on the error logs showing in the driver logs, we will understand what could be the cause of this error.

The error "Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetastoreClient" typically occurs when the Spark driver is unable to initiate a Hive client connection to the metastore. Is the 3306 port allowed on the Databricks VPC as per the docs?