sdp-meta (dlt-meta) vs lakeflow_framework: when should we use which?

[binlogreader]

Hi All,

I'm evaluating metadata-driven frameworks for Lakeflow Spark Declarative Pipelines, and I've found two that appear to serve the same purpose, both from Databricks organizations:

- databrickslabs/dlt-meta, whose feature/sdp-meta branch renames the project to sdp-meta

- databricks-solutions/lakeflow_framework repo

From reading both repos, here is what I can tell. They share the same problem statement: define bronze and silver pipelines through configuration instead of hand-written code, across batch and streaming, over the medallion architecture. sdp-meta is a Databricks Labs project where onboarding JSON or YAML becomes a DataflowSpec that a generic pipeline interprets at runtime, with a labs CLI and a Databricks App around it.

While lakeflow_framework comes from the solutions organization, describes itself as config-driven and pattern based, covers gold as well as bronze and silver, and is at v0.21.0 with support offered as best effort through GitHub issues.

What I can't work out from the outside is how to choose between them:

1. What purpose does each one serve that the other doesn't? They look like two answers to the same question, so I assume there is a reason both exist, maybe different origins or different target users.

2. When would you pick one over the other for a new project? For example, does one favor runtime interpretation of metadata while the other generates pipeline definitions, and does that change how upgrades, testing, or debugging feel day to day?

If anyone has run either of these in production, or has context on why the two exist side by side, I'd appreciate the perspective.

[anagilla]

One correction to the comparison: the released Labs project is dlt-meta v0.0.10; the sdp-meta v0.1.0 rename and lakeflow_framework's unified nodespec are still unreleased development work.

The released projects overlap because both build Spark Declarative Pipeline declarations from metadata, but their scope and metadata lifecycle differ. dlt-meta persists DataflowSpec records for Bronze/Silver ingestion, while lakeflow_framework loads and validates bundled specs across Bronze, Silver, and Gold; neither carries a Databricks support SLA.

1. For repeated Bronze/Silver ingestion, evaluate released dlt-meta v0.0.10 and account for a later migration after sdp-meta is tagged.

2. For Gold or dimensional/Data Vault/canonical modeling patterns, evaluate lakeflow_framework v0.21.0 using its released standard, flow, and materialized_view formats, not the open nodespec PR.

3. Before adopting either, compare direct Spark Declarative Pipelines (and Lakeflow Connect for supported ingestion), pin the framework version, and run workspace integration tests.

The pick mostly follows scope: if you stop at Silver, dlt-meta fits; if you need Gold and a modeling pattern, lakeflow_framework fits. If you can share which one you are targeting, I can point you at the more specific getting-started path.

[binlogreader]

Thanks for your prompt response. At present, we are focussed on ingesting data to silver. We are already using lakeflow connect for mysql db, so so we have the gateway, raw, bronze, and silver catalogs setup.

I would also be interested in gold since our downstream users work with that.

If you could point me to dedicated resources, that would be helpful for different groups!

[anagilla · Accepted solution]

Since you are already on Lakeflow Connect for MySQL, your CDC ingestion into Bronze is the managed, supported path, so you do not need a metadata framework for it. Resources by layer:

1. Ingestion to Bronze (what you have): stay on Lakeflow Connect. It lands destination streaming tables with optional SCD type 2, up to 250 tables per pipeline: MySQL connector, architecture.

2. Silver: your conforming and DQ logic sits downstream of Bronze. Build it as Spark Declarative Pipelines materialized views or streaming tables, or config-drive it across many tables with dlt-meta.

3. Gold (start here): SDP materialized views for dimensions, facts, and aggregates. Supported, no third-party framework needed: Spark Declarative Pipelines.

4. Gold (framework option): lakeflow_framework's Gold materialized-views pattern if you want config-driven dimensional or Data Vault modeling; it is pre-1.0, so pin a version (getting started).

For sources beyond MySQL later, dlt-meta is the entry point: docs, getting started.

[binlogreader]

Helpful direction, thank you!