Unity Catalog About Metastore

[tyorisoo]

Registered on 2024/10 from AWS marketplace.
We have created a customer management VPC and manually created the workspace.
No specific metastore settings were made when the workspace was created.
In the catalog screen of the account console,
unity catalog metastore is automatically created only by name.
Where can I find the metastore at this time?

[Alberto_Umana]

Hi @tyorisoo,

To find the Unity Catalog metastore that was automatically created, you can follow these steps:

- Account Console: Go to the Databricks account console. This is where you can manage various aspects of your Databricks account, including workspaces and metastores.

- Catalog Screen: In the account console, navigate to the catalog screen. This is where you should see the Unity Catalog metastore that was automatically created by name.

- Metastore Details: To get more details about the metastore, you can use the Databricks CLI. Here are the steps to retrieve the current assignment information for the workspace, including the metastore ID:

Set up the Databricks CLI using the guidance provided in the Databricks CLI documentation.
Run the following command to get the current assignment information:

databricks unity-catalog metastores get-assignment

This command will return details including the default_catalog_name, workspace_id, and metastore_id

[tyorisoo]

Thank you.
I checked with this query and it looks like I am getting the ID of the metastore
”select current_metastore()”
Does the metastore hold data?
If so, where is it managed?

[Alberto_Umana]

Hi @tyorisoo,

Please refer to this documentation: https://docs.databricks.com/en/data-governance/unity-catalog/get-started.html#enablement

When no specific metastore settings are made during the creation of a Databricks workspace, the data is not directly held by the metastore itself. Instead, the metastore manages metadata for data assets such as tables, views, and volumes within Unity Catalog. The actual data is stored in cloud storage locations specified in the metastore configuration.

If no metastore-level storage was configured, the data storage defaults to the storage locations defined at the catalog or schema level within the metastore. Each catalog or schema can have its own managed storage location, which is typically a path in a cloud storage account (e.g., AWS S3).

For workspaces that were enabled for Unity Catalog automatically, there is no metastore-level storage by default. Instead, a workspace catalog is created, and the managed location of this catalog is set to the storage account created for the workspace. This means that the data is stored in the cloud storage associated with the workspace catalog

[tyorisoo]

Thanks for getting back to me.

I have read the documentation.

The following from the documentation is applicable to the workspace I created.
1, A workspace catalog, which, when originally provisioned, is named after your workspace.
2, Your workspace gets the workspace catalog only if the workspace creator provided an appropriate IAM role and storage location during workspace 3, Automatic enablement of Unity.
3, Automatic enablement of Unity Catalog
　 An automatically-provisioned Unity Catalog metastore (unless a Unity Catalog metastore already exists for the workspace region and the metastore is enabled for automatic workspace assignment).

So after the workspace is created, the AWS root storage will have a unity-catalog folder as shown in the capture.
(I think the third thing you answered also corresponds to this capture)

There are a couple of points I don't understand.
1,Metastore manages metadata, not catalog information, schema information, table information, function information, access control information, etc.
In the current state, the metastore configuration is not done, so I can't see it? Can we see it without setting it in the metastore configuration from the account console?

2, When setting up in metastore configuration from the account console, will these information be stored in the target S3 bucket location?

  

[tyorisoo]

@Alberto_Umana
Sorry, I hope you are seeing this reply.

[Alberto_Umana]

Hello @tyorisoo,

I hope you are doing well!

Metastore manages metadata, not catalog information, schema information, table information, function information, access control information, etc. In the current state, the metastore configuration is not done, so I can't see it? Can we see it without setting it in the metastore configuration from the account console?

The metastore in Unity Catalog is responsible for managing metadata about securable objects such as tables, volumes, external locations, and shares, as well as the permissions that govern access to them. This functionality only applies if the metastore configuration is done. If the metastore configuration is not done, you will not be able to see the metadata. You need to configure the metastore from the account console to view and manage this information. Without setting up the metastore configuration, the metadata will not be visible.

When setting up in metastore configuration from the account console, will these information be stored in the target S3 bucket location?

When you set up the metastore configuration from the account console, you can specify a storage location for metastore-level storage of managed tables and volumes. This storage location is typically an S3 bucket in AWS. The metadata and other information managed by the metastore will be stored in the specified S3 bucket location. This setup ensures that the data is physically separated in storage and managed according to the configurations you provide during the metastore setup.