Hi All,
I'm evaluating metadata-driven frameworks for Lakeflow Spark Declarative Pipelines, and I've found two that appear to serve the same purpose, both from Databricks organizations:
- databrickslabs/dlt-meta, whose feature/sdp-meta branch renames the project to sdp-meta
- databricks-solutions/lakeflow_framework repo
From reading both repos, here is what I can tell. They share the same problem statement: define bronze and silver pipelines through configuration instead of hand-written code, across batch and streaming, over the medallion architecture. sdp-meta is a Databricks Labs project where onboarding JSON or YAML becomes a DataflowSpec that a generic pipeline interprets at runtime, with a labs CLI and a Databricks App around it.
While lakeflow_framework comes from the solutions organization, describes itself as config-driven and pattern based, covers gold as well as bronze and silver, and is at v0.21.0 with support offered as best effort through GitHub issues.
What I can't work out from the outside is how to choose between them:
1. What purpose does each one serve that the other doesn't? They look like two answers to the same question, so I assume there is a reason both exist, maybe different origins or different target users.
2. When would you pick one over the other for a new project? For example, does one favor runtime interpretation of metadata while the other generates pipeline definitions, and does that change how upgrades, testing, or debugging feel day to day?
If anyone has run either of these in production, or has context on why the two exist side by side, I'd appreciate the perspective.
