Bronze silver gold layers
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-15-2022 10:52 AM
Is there a best practise guide on setting up the delta lake for these 3 layers.
I'm looking for document or scripts to run that will assist me.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-15-2022 10:06 PM
Well the medallion architecture is not one fit for all use cases. It depends on your data landscape and how would you like to process data.
Bronze - Ingest your data from multiple sources.
Silver - Store clean and aggregated data
Gold - Store data to serve BI tools.
You need to design and implement your own pipeline for your own use case.
There are some small projects available on github which you can use to mimic the workflow. But their capability is very limited and they're far away from what real life data would look like.
This github repo and associated video can help you.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-05-2022 05:33 PM
hi @Lloyd Vickery ,
I would highly recommend to use Databricks Delta Live Tables (DLT) docs here https://databricks.com/product/delta-live-tables