anagilla
Databricks Employee
Databricks Employee

Since you are already on Lakeflow Connect for MySQL, your CDC ingestion into Bronze is the managed, supported path, so you do not need a metadata framework for it. Resources by layer:

  1. Ingestion to Bronze (what you have): stay on Lakeflow Connect. It lands destination streaming tables with optional SCD type 2, up to 250 tables per pipeline: MySQL connector, architecture.

  2. Silver: your conforming and DQ logic sits downstream of Bronze. Build it as Spark Declarative Pipelines materialized views or streaming tables, or config-drive it across many tables with dlt-meta.

  3. Gold (start here): SDP materialized views for dimensions, facts, and aggregates. Supported, no third-party framework needed: Spark Declarative Pipelines.

  4. Gold (framework option): lakeflow_framework's Gold materialized-views pattern if you want config-driven dimensional or Data Vault modeling; it is pre-1.0, so pin a version (getting started).

For sources beyond MySQL later, dlt-meta is the entry point: docs, getting started.

View solution in original post