cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Lakehouse Concept

User16826994223
Honored Contributor III

I want to understand lake house concept in very brief If I have to pitch for a customer in 1 minute

2 REPLIES 2

User16826994223
Honored Contributor III

Lakehouse is a concept defined with the following Parameter-

  1. Data is stored in an open standard format.
  2. Data is stored in a way which support Data Science,ML and BI loads.
  3. Delta is just a way or engine on cloud storage that provides control on data and prevent it from becoming data swamp and also add performance and provide sql like query support
  4. for lake house it is always recommended to have 3 layers,
  • Bronze - Raw data as it is from OTP
  • Silver -data in a curated format and with a filter that does not allow any junk data to silver, this layer is best suited for Data science and ML
  • gold layer-Purely aggregated data that helps in BI and can be used in Machine learning too.

Using the metastore for data lakehouse I have several follow up questions to your answer:

  1. Will companies generally have only one database that represents the data lakehouse?
  2. Will bronze tables be kept in the above database or will it have a separate database just for source data, or no database assignment at all?
  3. Files brought into as bronze tables will converted to Delta/Parqette?
  4. Will tables created in Silver tier be named with silver as a prefix or suffix, and if not how can we differentiate the silver tables from the gold?

I have not seen the best practice naming conventions anywhere.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group