cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Enable non-DE teams to build their own pipelines

PumpItUpper
New Contributor II

 

Hi! In all the latest dbks events I keep hearing this idea of shifting away from a centralized data team, and enabling the final consumer teams to build their own data pipelines with Lakeflow instead. This seems the worst idea ever to me, and sounds like dbks just trying to push Lakeflow to be honest, but I'm trying to keep an open mind and genuinely consider if this is a viable option long term.

Most people that have never done Data Engineering have no idea about DE concepts... which is understandable of course, but its still a fact that we have to deal with. Lets assume a simple classic setup for the sake of discussion:

  • Centralized DE team that provides a medallion architecture
  • Analysts that do dashboards and ad-hoc analytics notebooks on these tables
  • Machine Learning team that uses these tables for their models

Analysts and ML engineers might be amazing at their respective jobs, but in general they don't know anything about DE. They don't know about Data Contracts, CDC, SCD2, watermarking, window functions, etcetcetc. They have no idea what to do if there is a failure, how to ensure downstream propagation of changes, how to deal with corrupted data or checkpoints, etc...
Even if the coding is 100% done by AI, they still need to know the existence of these concepts at least in order to guide the AI.

You are telling me we should enable these people to build their own data pipelines? Ensuring they don't is one of my highest priorities right now.

Maybe I can see it working for a super simple batch use case that only appends data without ever changing anything, is never late, doesn't have PII-split, etc, but in a more complex case (any real case basically) I don't see this working out long term. In a big company its inevitable that whatever they create will eventually be used by other teams, who will in turn create their own stuff based on it...  this 2nd team will also have the ability to create downstream pipelines from what the Analysts/ML-engs created? Now we have a stack of processes that are dependent on stuff that wasn't built with the proper guardrails, all made by different people, none of which are specialists in this, and most likely don't even have communication with each other (because its different teams doing each part of the process).

I could go on, there are more issues I see with this, but what I said are my main grievances. I understand that a centralized team causes everyone to become dependent on them, and creates delays, but it still seems by far the better option long term for everyone involved.

2 REPLIES 2

ethan_parmer
New Contributor III

I agree with most of your concerns. The risks you list are real. But in my experience the choice isnโ€™t โ€œcentral team builds everythingโ€ vs โ€œeveryone builds anything.โ€ Whatโ€™s worked for us is a paved road:

  • The platform team owns the hard parts. Ingestion, CDC, SCD2 and Bronze/Silver stay with data engineering (Lakeflow Connect, AUTO CDC). Those are the pieces analysts shouldnโ€™t have to understand.
  • Domain teams build on certified Silver. Gold tables, features and marts are where business knowledge matters most and the engineering risk is lowest.
  • Guardrails are built in, not added later. Custom Asset Bundle templates (databricks bundle init <your-template>) give every new pipeline the same structure, expectations and job settings. Merges go through GitHub PR checks. Each domain gets its own Unity Catalog catalog, owned by a group.
  • Contracts are enforced, not just written down. Pipeline expectations on the tables other teams consume, plus Unity Catalog lineage and tags (certified vs sandbox), so you can see who depends on what before anything breaks.

In regulated life sciences we end up here anyway. Validated data flows go through change control, and exploratory work lives in a sandbox tier that canโ€™t feed validated outputs. That split happens to answer your dependency concern too.

Ethan

DoTA
Honored Contributor

Your worry is fair, but I'd push on where the dividing line sits. I don't think it should be team role (DE vs analyst). I think it should be how many other people depend on the output.

A table that only one analyst reads can be as fragile as they like, and the cost of that stays with them. The mess you describe starts when a second team builds on top of it. So make that moment visible: system.access.table_lineage tells you when something in a personal or sandbox schema picks up a reader from another team. Use that as the trigger to promote it into a certified schema where DE owns it, or at least reviews it, before it spreads further.

The other test I like is who carries the pager. If the team that wrote the pipeline is also the one paged when it breaks, they learn about checkpoints and late data quickly and tend to ask for help early. If nobody is willing to carry the pager for a pipeline, that's a good sign it shouldn't have been self-serve in the first place.

So: self-serve for low blast radius work, a promotion gate for anything shared. It also keeps your DE team's time for the hard parts instead of every small mart.