cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

What is the best practice for data model in silver layer in lakehouse

param_sen
New Contributor II

As per databricks https://www.databricks.com/glossary/medallion-architecture silver layers typically represent the "enterprise view" with improved quality than bronze (cleansed, deduplicated , augmented ) and mostly has 3NF like normalised data . 

The question is, do we really need to make it normalised if my raw data is flat? This will make room for more expensive joins when creating gold tables or materialized views in the gold layer. Analytics is one of the main consumers, and eventually there will be APIs that will serve the data in our use case.Any help will be greatly appreciated ?

 

1 REPLY 1

Palash01
Valued Contributor

Hey @param_sen 

Given your concerns about expensive joins and prioritizing analytics with flat raw data, here are some suggestions:

  1. Analyze the most common queries and reports you anticipate. Do they heavily rely on joins across dimensions? If not, the performance impact of normalization might be minimal.
  2. Prioritize denormalization for key joins: Strategically denormalize specific dimensions frequently joined with fact tables in the Gold layer. This can balance flexibility with query performance.
  3. Materialized views: Explore materialized views in the Gold layer to pre-join frequently accessed data, potentially mitigating some join costs.
  4. Begin with a less normalized Silver layer, focusing on cleansing and enriching the data.You can always introduce normalization later if needed.
Leave a like if this helps! Kudos,
Palash

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group