topic Bronze silver gold layers in Data Engineering

Bronze silver gold layers

BeginnerBob — Tue, 15 Mar 2022 17:52:29 GMT

Is there a best practise guide on setting up the delta lake for these 3 layers.

I'm looking for document or scripts to run that will assist me.

Re: Bronze silver gold layers

AmanSehgal — Wed, 16 Mar 2022 05:06:07 GMT

Well the medallion architecture is not one fit for all use cases. It depends on your data landscape and how would you like to process data.

Bronze - Ingest your data from multiple sources.

Silver - Store clean and aggregated data

Gold - Store data to serve BI tools.

You need to design and implement your own pipeline for your own use case.

There are some small projects available on github which you can use to mimic the workflow. But their capability is very limited and they're far away from what real life data would look like.

This github repo and associated video can help you.

Re: Bronze silver gold layers

jose_gonzalez — Wed, 06 Apr 2022 00:33:23 GMT

hi @Lloyd Vickery ,

I would highly recommend to use Databricks Delta Live Tables (DLT) docs here https://databricks.com/product/delta-live-tables