cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Databrick hive metastore location?

as999
New Contributor III

In databrick, where is hive metastore location is it control plane or data plane? for prod systems In terms of security what preventions should be taken to secure hive metastore?

8 REPLIES 8

Hubert-Dudek
Esteemed Contributor III
metastore_url = sc._jsc.hadoopConfiguration().get("javax.jdo.option.ConnectionURL")

It is also visible in cluster logs when the cluster is starting.

You can set your own metastore in Azure SQL or AWS RDS and connect it via a private link, so then it will be inside your infrastructure.

Hi,

I tried to run the above command in community notebook and the return result is "none". Is it that we can only give the url in paid version?

temp - Databricks Community Edition

295026
New Contributor II

i realized that this works in azure paid version but not in the community version

Also, may I know where to see the cluster logs when the cluster is starting, as you have suggested in the answer above?

as999
New Contributor III

@as999โ€‹ , Any thought in GCP databricks platform, workspace going to configured with Private cluster and database managed ip's, does the default metastore will reside private clusters GKE?

Atanu
Databricks Employee
Databricks Employee

If you are using internal metastore . yes, this will under databricks control plane (not exactly the control plane, but hosted in Databricks and managed by databricks). for external , it should as per your setup account.

Dhruv-22
New Contributor III

Hey Atanu
Can you point to any documentation reference that says that? Also, is this behaviour same for unity catalog metastore?

Prabakar
Databricks Employee
Databricks Employee

@as999โ€‹ The default metastore is managed by Databricks. If you are concerned about security and would like to have your own metastore you can go for the external metastore setup. You have the details steps in the below doc for setting up the external metastore.

AWS: https://docs.databricks.com/data/metastores/external-hive-metastore.html

GCP: https://docs.gcp.databricks.com/data/metastores/external-hive-metastore.html

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group