Questions about the design of bronze, silver, and gold for live streaming pipelines
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-01-2024 05:48 PM
I'm envisioning a live streaming pipeline.
The bronze, or data ingestion, is being fetched using the directory listing mode of the autoloader.
I'm not using File Notification Mode because I detect about 2-300 data changes per hour.
I'm thinking about implementing silver and gold tables later.
I tried to implement silver and gold as streaming tables, but it was not easy.
Do you usually implement silver and gold as a table or materialized view?
Also, can you introduce how to optimize when putting a large amount of data into redshift?