anagilla
Databricks Employee
Databricks Employee

One correction to the comparison: the released Labs project is dlt-meta v0.0.10; the sdp-meta v0.1.0 rename and lakeflow_framework's unified nodespec are still unreleased development work.

The released projects overlap because both build Spark Declarative Pipeline declarations from metadata, but their scope and metadata lifecycle differ. dlt-meta persists DataflowSpec records for Bronze/Silver ingestion, while lakeflow_framework loads and validates bundled specs across Bronze, Silver, and Gold; neither carries a Databricks support SLA.

  1. For repeated Bronze/Silver ingestion, evaluate released dlt-meta v0.0.10 and account for a later migration after sdp-meta is tagged.

  2. For Gold or dimensional/Data Vault/canonical modeling patterns, evaluate lakeflow_framework v0.21.0 using its released standard, flow, and materialized_view formats, not the open nodespec PR.

  3. Before adopting either, compare direct Spark Declarative Pipelines (and Lakeflow Connect for supported ingestion), pin the framework version, and run workspace integration tests.

The pick mostly follows scope: if you stop at Silver, dlt-meta fits; if you need Gold and a modeling pattern, lakeflow_framework fits. If you can share which one you are targeting, I can point you at the more specific getting-started path.