Databrick hive metastore location?

[as999]

In databrick, where is hive metastore location is it control plane or data plane? for prod systems In terms of security what preventions should be taken to secure hive metastore?

[Hubert-Dudek]

metastore_url = sc._jsc.hadoopConfiguration().get("javax.jdo.option.ConnectionURL")

It is also visible in cluster logs when the cluster is starting.

You can set your own metastore in Azure SQL or AWS RDS and connect it via a private link, so then it will be inside your infrastructure.

[irispan]

Hi,

I tried to run the above command in community notebook and the return result is "none". Is it that we can only give the url in paid version?

[295026]

i realized that this works in azure paid version but not in the community version

[irispan]

Also, may I know where to see the cluster logs when the cluster is starting, as you have suggested in the answer above?

[as999]

@as999​ , Any thought in GCP databricks platform, workspace going to configured with Private cluster and database managed ip's, does the default metastore will reside private clusters GKE?

[Atanu]

If you are using internal metastore . yes, this will under databricks control plane (not exactly the control plane, but hosted in Databricks and managed by databricks). for external , it should as per your setup account.

[Prabakar]

@as999​ The default metastore is managed by Databricks. If you are concerned about security and would like to have your own metastore you can go for the external metastore setup. You have the details steps in the below doc for setting up the external metastore.

AWS: https://docs.databricks.com/data/metastores/external-hive-metastore.html

GCP: https://docs.gcp.databricks.com/data/metastores/external-hive-metastore.html