Hi! In all the latest dbks events I keep hearing this idea of shifting away from a centralized data team, and enabling the final consumer teams to build their own data pipelines with Lakeflow instead. This seems the worst idea ever to me, and sounds like dbks just trying to push Lakeflow to be honest, but I'm trying to keep an open mind and genuinely consider if this is a viable option long term.
Most people that have never done Data Engineering have no idea about DE concepts... which is understandable of course, but its still a fact that we have to deal with. Lets assume a simple classic setup for the sake of discussion:
- Centralized DE team that provides a medallion architecture
- Analysts that do dashboards and ad-hoc analytics notebooks on these tables
- Machine Learning team that uses these tables for their models
Analysts and ML engineers might be amazing at their respective jobs, but in general they don't know anything about DE. They don't know about Data Contracts, CDC, SCD2, watermarking, window functions, etcetcetc. They have no idea what to do if there is a failure, how to ensure downstream propagation of changes, how to deal with corrupted data or checkpoints, etc...
Even if the coding is 100% done by AI, they still need to know the existence of these concepts at least in order to guide the AI.
You are telling me we should enable these people to build their own data pipelines? Ensuring they don't is one of my highest priorities right now.
Maybe I can see it working for a super simple batch use case that only appends data without ever changing anything, is never late, doesn't have PII-split, etc, but in a more complex case (any real case basically) I don't see this working out long term. In a big company its inevitable that whatever they create will eventually be used by other teams, who will in turn create their own stuff based on it... this 2nd team will also have the ability to create downstream pipelines from what the Analysts/ML-engs created? Now we have a stack of processes that are dependent on stuff that wasn't built with the proper guardrails, all made by different people, none of which are specialists in this, and most likely don't even have communication with each other (because its different teams doing each part of the process).
I could go on, there are more issues I see with this, but what I said are my main grievances. I understand that a centralized team causes everyone to become dependent on them, and creates delays, but it still seems by far the better option long term for everyone involved.