topic Re: Lakehouse Concept in Data Engineering

Lakehouse Concept

User16826994223 — Wed, 23 Jun 2021 06:11:19 GMT

I want to understand lake house concept in very brief If I have to pitch for a customer in 1 minute

User16826994223 — Wed, 23 Jun 2021 06:11:32 GMT

Lakehouse is a concept defined with the following Parameter-

Data is stored in an open standard format.
Data is stored in a way which support Data Science,ML and BI loads.
Delta is just a way or engine on cloud storage that provides control on data and prevent it from becoming data swamp and also add performance and provide sql like query support
for lake house it is always recommended to have 3 layers,

Bronze - Raw data as it is from OTP
Silver -data in a curated format and with a filter that does not allow any junk data to silver, this layer is best suited for Data science and ML
gold layer-Purely aggregated data that helps in BI and can be used in Machine learning too.

gregc001 — Wed, 02 Nov 2022 21:28:13 GMT

Using the metastore for data lakehouse I have several follow up questions to your answer:

Will companies generally have only one database that represents the data lakehouse?
Will bronze tables be kept in the above database or will it have a separate database just for source data, or no database assignment at all?
Files brought into as bronze tables will converted to Delta/Parqette?
Will tables created in Silver tier be named with silver as a prefix or suffix, and if not how can we differentiate the silver tables from the gold?

I have not seen the best practice naming conventions anywhere.