topic Re: Databrick hive metastore location? in Data Engineering

Databrick hive metastore location?

as999 — Sat, 02 Apr 2022 15:59:09 GMT

In databrick, where is hive metastore location is it control plane or data plane? for prod systems In terms of security what preventions should be taken to secure hive metastore?

Re: Databrick hive metastore location?

Hubert-Dudek — Sat, 02 Apr 2022 16:31:23 GMT

metastore_url = sc._jsc.hadoopConfiguration().get("javax.jdo.option.ConnectionURL")

It is also visible in cluster logs when the cluster is starting.

You can set your own metastore in Azure SQL or AWS RDS and connect it via a private link, so then it will be inside your infrastructure.

Re: Databrick hive metastore location?

as999 — Sat, 02 Apr 2022 21:01:34 GMT

@as999 , Any thought in GCP databricks platform, workspace going to configured with Private cluster and database managed ip's, does the default metastore will reside private clusters GKE?

Re: Databrick hive metastore location?

Atanu — Fri, 13 May 2022 05:58:44 GMT

If you are using internal metastore . yes, this will under databricks control plane (not exactly the control plane, but hosted in Databricks and managed by databricks). for external , it should as per your setup account.

Re: Databrick hive metastore location?

Prabakar — Wed, 18 May 2022 09:58:00 GMT

@as999 The default metastore is managed by Databricks. If you are concerned about security and would like to have your own metastore you can go for the external metastore setup. You have the details steps in the below doc for setting up the external metastore.

AWS: https://docs.databricks.com/data/metastores/external-hive-metastore.html

GCP: https://docs.gcp.databricks.com/data/metastores/external-hive-metastore.html

Re: Databrick hive metastore location?

irispan — Mon, 22 Aug 2022 14:47:50 GMT

Hi,

I tried to run the above command in community notebook and the return result is "none". Is it that we can only give the url in paid version?

Re: Databrick hive metastore location?

irispan — Mon, 22 Aug 2022 14:48:43 GMT

Also, may I know where to see the cluster logs when the cluster is starting, as you have suggested in the answer above?

Re: Databrick hive metastore location?

295026 — Wed, 24 Aug 2022 05:25:23 GMT

i realized that this works in azure paid version but not in the community version

Re: Databrick hive metastore location?

Dhruv-22 — Sat, 18 May 2024 07:50:24 GMT

Hey Atanu
Can you point to any documentation reference that says that? Also, is this behaviour same for unity catalog metastore?