- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thursday - last edited Thursday
Since you are already on Lakeflow Connect for MySQL, your CDC ingestion into Bronze is the managed, supported path, so you do not need a metadata framework for it. Resources by layer:
-
Ingestion to Bronze (what you have): stay on Lakeflow Connect. It lands destination streaming tables with optional SCD type 2, up to 250 tables per pipeline: MySQL connector, architecture.
-
Silver: your conforming and DQ logic sits downstream of Bronze. Build it as Spark Declarative Pipelines materialized views or streaming tables, or config-drive it across many tables with dlt-meta.
-
Gold (start here): SDP materialized views for dimensions, facts, and aggregates. Supported, no third-party framework needed: Spark Declarative Pipelines.
-
Gold (framework option): lakeflow_framework's Gold materialized-views pattern if you want config-driven dimensional or Data Vault modeling; it is pre-1.0, so pin a version (getting started).
For sources beyond MySQL later, dlt-meta is the entry point: docs, getting started.