cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Three level name space naming standard

cltj
New Contributor III

Hi all, I have not been successful in getting a good grip of the naming conventions for the three level name space. 
Initially i learned about bronze, silver and gold, but i am confused where to put this. The obvious choice may be to use the {catalog} top level for this, but how will this scale when new usecases and sorces and sensitive data comes in, and ultimately how will it impact access management? 

I am looking for how others have their naming standards and how they solve for the questions i pose above. I am also looking for others comments on my attached picture. 

Thank you in advance. 

3 REPLIES 3

cltj
New Contributor III

Thanks for your prompt response.
I dont know why it does not show up for the preview. But downloading it worked for me. 

@Retired_mod 

Outlier_Talks
New Contributor

Hey , Glad you raised the confusion at correct place .

It's pretty common mistake to deep dive into Data transformations etc..when you have data .But it's really important to understand where your data resides and how it's being governed.

I recommend you to check my LinkedIn post on how to make use 3 level name space wisely: https://lnkd.in/p/gntZW2u3

 

rkhand14_ltm
Databricks Partner

@cltj 

This is a classic example of how an architecture can become difficult to scale over time. Initially, we onboarded only a few clients and maintained client-specific tables. However, as the number of clients grew, the solution resulted in a large proliferation of tables, increasing maintenance and operational complexity.

To address this, we redesigned the architecture using a metadata-driven framework powered by control/metadata tables. The control/metadata tables store column mappings and transformation rules, while a centralized transactional table stores common attributes along with sequence numbers and source identifiers (Like - sql, excel, datavserse).

This approach significantly improved scalability and maintainability. Onboarding a new client now requires only configuration changes in the metadata layer, without creating additional physical tables. Client-specific views can be exposed at the Gold layer, enabling easy access to curated data while keeping the underlying architecture standardized and scalable.

Naming convention should be like below:   

department.Usecase.layer  =   

dep.finance.bronze_meta  and dep.finance.bronze_transactional

dep.finance.silver_meta  and dep.finance.silver_transactional

dep.finance.gold_meta  and dep.finance.gold_transactional , if require client specific view on the basis of sources.