cancel
Showing results for 
Search instead for 
Did you mean: 
Data Governance
Join discussions on data governance practices, compliance, and security within the Databricks Community. Exchange strategies and insights to ensure data integrity and regulatory compliance.
cancel
Showing results for 
Search instead for 
Did you mean: 

Unity Catalog & Delta Lake

Tram
New Contributor III

In setting up Databricks accounnt, we need to provide storage (s3 buckets) to store the final cleaned data or aggregated data. If the Unity Catalog is enabled at the beginning of account set up, we'd need to to provide S3 buckets again to store the metedata. My question: can it be one S3 bucket for final cleaned data and the Unity Catalog, or do they need to be separate s3 buckets?

3 REPLIES 3

Pat
Honored Contributor III

HI @Tram Nguyen​ ,

I am not sure if I got this correctly but I believe you are referring to the:

  • Root storage for a workspace: Root storage for workspace objects like cluster logs, notebook revisions, and job results libraries (https://docs.databricks.com/administration-guide/account-api/aws-storage.html#configure-aws-storage)
  • Metastore storage: Each metastore is configured with a root storage location in an S3 bucket in your AWS account. This storage location is used for metadata and managed tables data.(https://docs.databricks.com/data-governance/unity-catalog/index.html#metastores)

It's not recommended to re-use the workspace s3 bucket for metastore.

image 

thanks,

Pat.

karthik_p
Esteemed Contributor

@Tram Nguyen​ above pat provided links will have clear information. but to provide overview

  1. account s3 bucket should be different
  2. unity catalog s3 bucket should be different (Meta store in technical terms)

unity catalog bucket should be configured in same region of your account, so that no network Issues will be seen. multiple workspaces in that account can use same unity catalog metastore (which is s3 bucket)

Kaniz_Fatma
Community Manager
Community Manager

Hi @Tram Nguyen​, We haven’t heard from you since the last response from @Pat Sienkiewicz​ and @karthik p​​, and I was checking back to see if their suggestions helped you.

Or else, If you have any solution, please share it with the community, as it can be helpful to others.

Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!