- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-10-2025 06:00 AM
Our initial approach was to have catalogs for sources and uses, but that was confusing and the grade of data wasn't obvious. And, as we began to join data, it was unclear where to put the final datasets. We switched to three catalogs--one for bronze, one for silver, one for gold. This has made life so much less confusing for us, not to mention the systems which rely on our data.
In bronze we have a schema for each source. The landed data is in an external volume, and we flatten the data into tables in the same schema. Silver has schemas for different entities we're working with, like "person" or "product". Gold we have schemas for different consumers or departments, there isn't a lot of overlap in data. This way we can restrict end users to specific gold schemas.