cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Lakehouse Concept

User16826994223
Honored Contributor III

I want to understand lake house concept in very brief If I have to pitch for a customer in 1 minute

2 REPLIES 2

User16826994223
Honored Contributor III

Lakehouse is a concept defined with the following Parameter-

  1. Data is stored in an open standard format.
  2. Data is stored in a way which support Data Science,ML and BI loads.
  3. Delta is just a way or engine on cloud storage that provides control on data and prevent it from becoming data swamp and also add performance and provide sql like query support
  4. for lake house it is always recommended to have 3 layers,
  • Bronze - Raw data as it is from OTP
  • Silver -data in a curated format and with a filter that does not allow any junk data to silver, this layer is best suited for Data science and ML
  • gold layer-Purely aggregated data that helps in BI and can be used in Machine learning too.

Using the metastore for data lakehouse I have several follow up questions to your answer:

  1. Will companies generally have only one database that represents the data lakehouse?
  2. Will bronze tables be kept in the above database or will it have a separate database just for source data, or no database assignment at all?
  3. Files brought into as bronze tables will converted to Delta/Parqette?
  4. Will tables created in Silver tier be named with silver as a prefix or suffix, and if not how can we differentiate the silver tables from the gold?

I have not seen the best practice naming conventions anywhere.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.