Does your “open source” architecture actually deliver freedom? For an enterprise, lock-in isn’t a binary choice; it is a spectrum of exit costs that must be weighed against total TCO—specifically the hidden weight of OPEX. This piece aims to challenge the common narrative, exploring how open source can generate significant operational gravity while certain commercial stacks remain surprisingly portable.
[Disclaimer: the opinions expressed in this article are my own and do not represent any official statement from my employer]
I have been doing this for more than a decade as a Consultant on big data, Cloud, and Platform architectures, migrations, governance, and now AI … the whole catalog: every other one or two quarters, someone in a meeting opens a slide with two columns. "Open source" on the left, green checkmarks ✅. "Proprietary" on the right, red crosses ❌. The conclusion is supposed to be obvious.
It isn't. The slide is wrong, or at least it is the wrong slide.
Having spent a lot of time in my career advocating for "lock-in-free architectures" across diverse industries, teams, tech stacks, and cloud environments, I now possess a comprehensive view of this dynamic. Joining Databricks has now also provided me with a full product and portfolio lens. Consequently, I can clearly see both perspectives: the business side, which battles the fear of lock-in while balancing operational costs, talent shortages, immediate time-to-market, and ROI; and the vendor side, which concentrates on integrating an expanding catalog of products into efficient, seamless, and rapidly deployable production frameworks for the enterprise.
The purpose of this article is to clarify, with examples and references, what I personally believe is a misconception, as per the title:
Open Source Doesn't (Necessarily) Mean You're Free. Closed Source Doesn't (Necessarily) Mean You're Locked-in.
I also want to thank my network of present and former colleagues for the intense and insightful conversations we have more and more often on this topic, which inspired me to write this piece.
---
So, we were saying that Open Source doesn’t automatically mean freedom, just like Closed Source doesn’t automatically mean a cage. What actually decides where you end up are the things nobody puts on the slide:
- how much does it cost to integrate this tool into the rest of your stack today, and
- how much does it cost to leave when the time comes? Because the time comes. Always.
Let’s elaborate together on these two points.
A quick word on definitions
The Open Source Initiative has a very specific definition of open source. The license must give you the source code, plus the right to modify and redistribute it for any purpose, including commercial use. Important and useful. But notice what the definition itself does not hand you: a healthy community, a sane upstream roadmap, more than one significant maintainer, or any of the integrations you actually need. The license tells you what you are allowed to do. It says nothing about what you can afford to do.
Lock-in has its own clean definition: a situation in which switching to another vendor incurs substantial switching costs. That word "substantial" is doing a lot of work and the costs you need to consider come from a long list of sources: proprietary data formats, APIs, and tooling. But it can also include the integrations your team built around the product, the skills your engineers have acquired, and the downstream pipelines and consumers that depend on the current behavior. Only some of that has anything to do with whether the source code is on GitHub.
I have watched open source lock people in
You might already have seen this happen without noticing: open source can also lock people and companies in. The most frequent ways for this to unfold are, in my experience:
- Relicensing: the project starts with an open license, gets traction, and then transitions into a commercial one
- Operational Gravity: an open source tool or framework still needs to be installed, operated, and maintained by somebody in your organization
- Proprietary surfaces inside (and outside) open source: when the surrounding container is open source, but some internals are proprietary
- A combination of (open source) tools: the compounding effect of operational gravity multiplied by the number of tools to combine and integrate together
Let’s analyze these points one by one in detail with examples.
Relicensing
You pick a once-permissive project precisely because it is open, then watch the license change under your feet a few years later. All the examples below sit firmly in the data, streaming, or AI territory:
- MongoDB moved its Community Server from AGPL to the SSPL in October 2018. The OSI never recognized SSPL as an open source license, and MongoDB withdrew its submission for approval in 2019.
- Confluent relicensed KSQL, Schema Registry, REST Proxy, and several Confluent Platform connectors from Apache 2.0 to the Confluent Community License in December 2018. Apache Kafka itself stayed under the Apache 2.0 license at the ASF. Still, anyone who had standardized on the Confluent distribution discovered that parts of what they relied on now had explicit hosting restrictions.
Just to cite a few, but there are stories about Elasticsearch and Kibana, Redis, Terraform, and many others, all following the same pattern: in each of these cases, customers who had picked those projects to escape vendor concentration ended up with exactly the problem they were trying to dodge. The OSS community sometimes reacted to these relicensing operations with OSS forks (e.g., OpenTofu fork of Terraform), thus theoretically “solving” the licensing problem, but… solving the lock-in one? Are community-driven forks of mainstream projects really worth being the foundation of your next Enterprise Data & AI strategy?
Operational gravity
Self-hosting Kafka, Airflow, Spark, Iceberg, dbt, and friends is legitimate, but has a drawback: it implies having also a multi-engineer platform team, a release-coordination calendar, and a thick pile of internal documentation that exists nowhere else in the world - because it’s your custom-tailored and integrated stack. The licenses are free; the engineers, runbooks, upgrade choreography, security hardening, and on-call rotations are not. The dominant “live” costs of an open source stack live in:
- Integration
- Customization
- Maintenance (bug fixes - are you sure you can fix yourself, or wait until it’s fixed and released in OSS? Upgrades, especially major ones - they may require a significant amount of work)
- On-call
Vendor invoices become payroll.
I have seen this so many times: the OPEX costs for the platform team were 5 to 10x the infra costs, delivery was not fast enough, and fear of change was high due to re-wiring needs. Then, with people turnover and legacy knowledge fading, what was a no-lock-in quickly became a legacy-knowledge-no-innovation-slow-delivery lock-in.
Proprietary surfaces inside open source projects
This one is more subtle. Take ClickHouse: it is Apache 2.0, so the license checkbox is firmly green. But the on-disk format used by the MergeTree engine is understood only by ClickHouse itself: not Parquet, not Iceberg, not Delta. ClickHouse can read and write Parquet at the edges. Still, its native storage format is not an open standard, which means the day you choose to query the same data from Spark, Trino, DuckDB, or any cloud warehouse, you are not flipping a config: you are running an ETL job to convert (read: duplicate somewhere else) every byte you ever wrote.
On top of that, fine-grained access control, row and column security, lineage, audit, data quality, and data catalog are not first-class enterprise concerns of the project, so you bolt on Apache Ranger or Apache Polaris or roll your own - more glue, more on-call, more lock-in to your own glue.
The license is open. The data and the surrounding control plane are not free to leave.
Another example could be Apache Cassandra vs. the commercial offering of DataStax Enterprise (DSE): while the former provides a basic open source core, you are forced to rely on the commercial counterpart if you want enterprise-grade features like advanced security, replication, indexed search, and other proprietary-only extensions. Not to mention the enterprise support.
"Talk-to-your-data" lock-in
Let’s take the “Talk-to-your-data” use case as an example, as it’s the freshest, and the one I see eating the most calendar lately. The brief sounds innocent: "Let business users ask questions in natural language and get answers from our data." Half a sentence on a slide. Six months of engineering.
You wire up an open source text-to-SQL agent stack. A vector store for schema documentation, sample queries, and the business glossary. The embedding model of the quarter. An orchestration framework. A schema-linking (semantic) layer that maps natural-language entities to columns. A reranker. Prompt templates you tune over weeks of trial and error. A handful of MCP servers wrapping your warehouse(s) and metric layer. A SQL-validation pass. A hallucination guard. An evaluation harness on top of MLflow or LangFuse. A homegrown agent loop holding the whole thing together. Every component is open source. Every component is, on paper, replaceable.
Then the fun begins.
Change the embedding model, and every chunk of schema documentation, every certified query, and every glossary entry has to be re-embedded and re-indexed; embedding spaces are model-specific and not interoperable across providers.
Change the orchestration framework and your chains, retrievers, callbacks, and tool definitions all change shape; your code does not port over unmodified.
Change the vector store and ingestion, the indexing strategy, and the latency budget; all get redesigned.
Change the underlying LLM and prompts; evals and tool-calling formats also need retuning. None of these projects is closed.
The lock-in lives in the combination, in the embedding you committed to, in the schema-linking heuristics you wrote, and in the bespoke glue between every layer. Your team owns all of it. Forever.
And after those six months and the work of 4 people, you have a "talk to your data" agent that probably works on twenty tables for the first 3 months. The business has thousands. Hundreds more every other quarter.
Managed Service/Closed source can be surprisingly portable
Now flip it. A commercial managed product that stores its data in open formats, exposes open APIs, releases parts of its products as open source to foster an ecosystem of possible native integrations, and integrates cleanly with what you already run via open standards/protocols is, in practice, easy to leave. It’s not always the same for every commercial product, of course, that’s why you must choose wisely.
Your data sits in Delta Lake or Iceberg for OLAP, and in Postgres for OLTP. Your queries are plain SQL. The accumulated engineering expertise is not necessarily locked to the specific vendor. I will be honest, Databricks is a commercial platform and has its own proprietary surfaces, like Photon, runtime optimization on top of Spark, some unique Unity Catalog or Open Sharing capabilities, and its own BI layer.
The point of this article is not “Databricks is open and therefore free of lock-in”.
The point is that the data lives in open formats (Delta, Iceberg, Postgres), the query language is SQL, and the engineering skills (Spark, Python, MLflow, Unity Catalog) can be transferred to other platforms.
Those are the surfaces that determine the cost of leaving. Proprietary runtime optimizations are a tradeoff customers can knowingly accept in exchange for performance and operational simplicity, but they should accept them knowing what they are.
The day you decide to change it, you change it. Painful? Yes. But the data is already in an open format another engine can read, the queries are already SQL, another engine can run, and the skills your team built, e.g., SQL, Python, Spark, MLflow, transfer to whatever comes next. The switching cost is real and unavoidable (by the way, regardless of your stack!), but it lies in the migration project, not in the data or people layers. That’s a different cost category than rebuilding a hand-assembled OSS stack from scratch.
The two-column slide hides where actually lock-in lives in:
- storage (formats, location, accessibility)
- integration capabilities / available APIs
- accessibility to tech AND business users
- slowness of innovation
- high OPEX
- lack of talent availability
- legacy knowledge
- and “isolated open silos”
Migrations happen on a clock
Here is the bit architects know and PowerPoint forgets: enterprises migrate. Constantly.
Hardware refresh cycles run 3 to 5 years for compute and storage, 5 to 7 for networking. Application portfolios churn faster than those under the joint pressure of cloud, AI, and regulation. New trends and opportunities show up, new tools are released every quarter, and a new integration need is at the door in another month. It’s said that enterprise software applications become legacy the day after they are released.
The takeaway is the same regardless of which study you trust: you are never not migrating. There will always be a "legacy platform" being decommissioned, a "current one" being optimized and integrated with newer stuff, and a "future one" being evaluated. Which means that when you make an architecture decision today, you are not picking the tool you will use forever: you are picking the tool whose cost of leaving will land on somebody's desk in three to seven years.
YES, that somebody might be you. The same one today has to answer the business why it takes so long to get access to the data, to query it with natural language, or to integrate a new data source.
So an important strategic question (at least, when drafting an enterprise Data & AI strategy/architecture) should be, in my opinion:
Over the next five to seven years, how much of my team's calendar does this choice consume, first to integrate, then to operate, and eventually to migrate?
BONUS: And what will my competitors do in the meantime?
The full-stack tax
Also, please, stop thinking about one tool. Picture instead the actual shopping list for a modern enterprise data and AI platform, and decide for each layer whether you build it, buy it, or assemble it from open source projects.
The list is not short:
- Object storage and table format
- Distributed computing for batch and streaming. ETL and ELT
- A SQL query engine
- Orchestration
- Catalog with lineage tracking and auditing
- Security layer for fine-grained access control
- Data quality
- Identity management and SSO
- Machine Learning models training, experiment tracking, model registry, model serving, model monitoring.
On top of all that, the new generative-AI surface:
- Agents (and their memory, guardrails, orchestration)
- LLMs (and their usage, observability, monitoring, cost management)
- RAG
- Vector Search
- "Talk to your data" interfaces
- Applications
- Skills, MCP servers, and the prompt registry around them.
- A BI/Reporting layer
- Private connectivity all over
And of course, observability and monitoring for every one of those layers, plus cost attribution (FinOps), plus disaster recovery (sometimes Cross-Cloud).
You can build every single one of those layers from open source projects.
The names are well-known: Iceberg, Delta, Spark (with on top - you can’t hide it - your own custom YAML-based ingestion framework), Flink, Kafka, Airflow, dbt, Trino, ClickHouse, MLflow, LangFuse, Ray, KServe, vLLM, Apache Polaris, Apache Ranger, OpenMetadata, OpenLineage, Marquez, Prometheus, Grafana, Keycloak, OPA, LangGraph, MCP servers, plus whichever "natural-language-to-SQL" library is in fashion this quarter. None of these is a bad project. Some are excellent.
The question is what happens when you put all of them in the same room and tell them to behave like one platform.
Three things, all of which appear as line items in the budget the year after the architecture decision was approved.
First, integration code. Each project is internally coherent and externally indifferent. Identity models do not match. Permission models do not match. Metadata models do not match. Lineage stops at the boundary of each tool. Quality rules are expressed in three different ways. Audit events live in three different sinks. To make any of this look like one platform, somebody has to write and own the bridges, and that somebody is your team.
Forever.
Second, release coordination. Every project ships on its own clock. Compatibility matrices across a dozen projects are a small combinatorial nightmare. A non-trivial fraction of platform engineering on an OSS-only stack is making sure that next quarter's Spark, Iceberg, Polaris, Ranger, Airflow, dbt, and MLflow versions still play nicely together. None of that work is visible to the business, but "maintenance service interruption windows" to upgrade these tools.
And you also do this forever.
Third, expertise scarcity. The single Spark expert. The single Airflow expert. The single MLflow operator. The one engineer who keeps the Polaris-and-Ranger glue alive. Each of these people is a single point of failure for a layer of your stack, and the labor market for replacing any of them is brutal. The software is free. The expertise is not, and please don't fall into the trap of thinking "AI will do it for me" - at least not yet.
This is a forever concern as well.
Add the three together, and what you actually get is a longer time-to-market, because every business request now has to pass through that integration layer before it can ship, and an overall higher Total Cost of Ownership (TCO), because the integration layer never stops needing care and feeding. Both costs are paid out of the same budget that was supposed to fund "creating business value".
Leaving open source is its own migration
Here is the part that catches people out. The instinct says, "If I ever want to move, I have the source code, I can move". In practice, leaving a hand-assembled open source stack can be as painful as leaving a managed service or Microsoft Excel, sometimes more so, for three concrete reasons:
- The data formats are not always portable. ClickHouse MergeTree, untouched Hive directory layouts, MLflow runs persisted into project-specific schemas: all of those need conversion before any successor platform can read a byte. Having the source code does not change the bytes on disk.
- The integration code is bespoke. The bridges between Airflow, Ranger, Polaris, dbt, OpenMetadata, and your custom RBAC layer are documented exactly as well as your team had time to document them, which is to say, partially and - let’s be honest - not always up to date. Whoever inherits the migration has to reverse-engineer that code from scratch, and they get to do it without the people who originally wrote it.
- The skills are stack-specific. The engineer who is fluent in Kafka, your Spark physical plan, your Airflow DAGs, and your custom authentication layer cannot just transfer that fluency to a different stack. Retraining time is a real cost, and it is paid in calendar quarters, not weeks.
I know what you're thinking... YES, it would be similar in the case of migrating from one vendor to another… so we do agree on the title of this section, right? A product can do the integration work for you during the build years and keep your data and skills portable during the leave years. Those are the two axes that matter.
A better slide
If I had to redraw the comparison, I would retire "open vs. closed" entirely and put two questions on screen instead.
A tool can be fully open source and score badly on both questions. A commercial product can score well on both. The license or cost of the managed service is one input among many. An integrated portfolio can dramatically lower integration cost during the build years, and that's a real economic benefit worth paying for. But that is not the same thing as being free to leave. What determines whether you're locked in is whether your data, formats, and skills travel with you when you choose to migrate. Choose vendors that score well on both axes; integration cost is low, exit cost is low, and be honest that those are two separate questions.
This section could be extended far longer, as taking architectural decisions is an art, refined over decades of experience, but also supported by tools and frameworks, better if quantitative (i.e., I know well it’s not just an open vs. closed comparison, but this article’s focus was on the demystification of this narrative).
For example, a more fair and robust approach I’ve sometimes used is the Weighted Decision Matrix framework or, especially when a vendor we have already adopted as part of our stack provides what we might need, the Pugh Matrix one to set the current platform/vendor as the benchmark baseline, then define the decision pillars that apply to that specific decision, i.e., aligned to the specific use case requirements.
Finally, quantify (absolute or relative quantities), optionally assign each pillar a weight, and obtain an overall score to drive the decision.
But I don’t want to go off-topic here, so I’ll leave it to the reader to dive deeper as a follow-up if necessary.
In the end
Open source matters. I have spent a decade using, deploying, and operating it, and I am not stopping. Databricks itself is built on open source, and it has a dedicated principle in its “Well Architected Framework” specific to interoperability and usability, which I believe are the real foundation of an “open (read: not locked-in) architecture”.
Furthermore, in the recent 1st day keynote of the 2026 Databricks Data + AI Summit (check out the recording, if you haven’t watched it), the Databricks CEO, Ali Ghodsi, presented a panel that resonated a lot with this article’s scope, I think:
I think the pillar “Choice” there on the right is what I’ve tried to unfold in this article, besides the more obvious, but not to underestimate - “Cost” one: no lock-in means having choices for interoperability, integration, migration, and evolvability. Today more than ever, you don’t want to be locked-in with one AI, one foundation model, one LLM vendor, one API, one vertical - I think we can agree on this at this stage - BUT the point is: the best alternative is not necessarily and only the full open source way.
“Control” is tightly coupled, because a messy integration among open source “free” tools lacks the enterprise-grade control and governance you are looking for (not necessarily when experimenting, but - trust me - BEFORE going into production with an Agentic system). “Context” is the glue that makes “n-tools-integrated-together” a real AI-ready, enterprise-grade platform, but this topic may be a deeper analysis for another article.
I’d like to wrap up recommending fellow architects and tech leaders to drop the religion, as the narrative "Open license equals freedom, closed license equals a cage" has been costing more team-years than almost any other myth in our industry. The next time the two-column slide shows up in a meeting, ask one question: over the next migration cycle, where exactly will the effort and the lock-in actually live?
The version of you running the migration five years from now will be grateful. The business users of today will be as well.
———
Roberto Coluccio is a Delivery Solutions Architect at Databricks. The views in this post are his own.
Thank you João "Jota" Martins, Rouzbeh Rahimi, and Alex Ott for the valuable review and comments.
