โ04-02-2022 08:59 AM
In databrick, where is hive metastore location is it control plane or data plane? for prod systems In terms of security what preventions should be taken to secure hive metastore?
โ04-02-2022 09:31 AM
metastore_url = sc._jsc.hadoopConfiguration().get("javax.jdo.option.ConnectionURL")
It is also visible in cluster logs when the cluster is starting.
You can set your own metastore in Azure SQL or AWS RDS and connect it via a private link, so then it will be inside your infrastructure.
โ08-22-2022 07:47 AM
โ08-23-2022 10:25 PM
i realized that this works in azure paid version but not in the community version
โ08-22-2022 07:48 AM
Also, may I know where to see the cluster logs when the cluster is starting, as you have suggested in the answer above?
โ04-02-2022 02:01 PM
@as999โ , Any thought in GCP databricks platform, workspace going to configured with Private cluster and database managed ip's, does the default metastore will reside private clusters GKE?
โ05-12-2022 10:58 PM
If you are using internal metastore . yes, this will under databricks control plane (not exactly the control plane, but hosted in Databricks and managed by databricks). for external , it should as per your setup account.
โ05-18-2024 12:50 AM
Hey Atanu
Can you point to any documentation reference that says that? Also, is this behaviour same for unity catalog metastore?
โ05-18-2022 02:58 AM
@as999โ The default metastore is managed by Databricks. If you are concerned about security and would like to have your own metastore you can go for the external metastore setup. You have the details steps in the below doc for setting up the external metastore.
AWS: https://docs.databricks.com/data/metastores/external-hive-metastore.html
GCP: https://docs.gcp.databricks.com/data/metastores/external-hive-metastore.html
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group