A few months back my team at Genpact picked up a real world evidence project for a life sciences client that had been running on Kedro for a couple of years. The pipeline touched patient level claims and EHR extracts, ran feature engineering for a handful of outcomes models, and trained models that fed into a reporting layer. It worked, but it had grown the way these things always do. New analysts kept adding nodes, config files multiplied across environments, and nobody outside the original team wanted to touch the DAG. When management asked us to consolidate everything onto Databricks, the first question wasn't whether Kedro was good or bad.
"It was how much of it we actually needed to keep."
Then came the final nail in the coffin, the Kedro Project was flagged at Client Governance monthly meet as a project which is using older versions of python libraries, and with many packages which are not supported by the latest version of python on serverless cluster setup. While the issue could be mitigated by packaging and wrapping the code in a Databricks Assets Bundle which needed another review round for compliance checks and permissions.
Why we didn't just rip it out
Kedro gave this project something worth preserving: pipelines that are testable, versioned, and easy to reason about node by node. Ripping that out and rewriting everything as raw PySpark scripts would have meant redoing months of validation work that regulatory reviewers had already signed off on. So the plan wasn't a rewrite. It was a migration where Kedro's pipeline structure stays as the mental model, but the runtime, storage, governance, and scheduling all move onto Databricks native services.
Kedro's own docs describe three ways to run a project on Databricks, and we ended up borrowing pieces from all three rather than picking just one, which the docs actually acknowledge is normal depending on where your team is in the project lifecycle.
The four pieces that mattered
GitHub as the source of truth. The Kedro repo stayed on GitHub, and we connected it into Databricks using Git folders instead of copying code into the workspace by hand. This kept every change to a node or a catalog entry reviewable through a pull request, and it meant the workspace was never the place where code actually lived. Databricks has a whole pattern for this called CI/CD with Git folders, including a production folder that only updates when a PR merges, which is exactly the discipline you want on a project touching PHI.
Serverless compute for the jobs themselves. We stopped provisioning and babysitting job clusters. Serverless compute for workflows lets Databricks handle scaling and node selection per task, and for a pipeline with uneven load between the ingestion nodes and the training nodes this removed a lot of guesswork around cluster sizing that used to eat up sprint time. It's also just less for a life sciences client's infra team to sign off on from a security review standpoint, since there's no persistent cluster configuration to audit. In addition by migrating we could remove legacy code to maintain hooks which keep an spark session active.
Unity Catalog for governance, not just storage. This was the part that actually mattered most given the PHI involved. Every Kedro dataset in the catalog.yml got remapped from local paths to Unity Catalog tables and volumes, which meant column level tagging, masking policies on direct identifiers, and row level filters became possible in a way that flat files in a Kedro data folder never allowed. Databricks has published a fair amount on this specifically for healthcare, and a lot of it lines up with basic HIPAA minimum necessary principles once you translate it into catalog language.
MLflow for the model side. The training pipeline still runs as a set of Kedro nodes, but every run now logs to MLflow, and the registered models live in the Unity Catalog model registry instead of a pickle file sitting in a shared drive. MLflow made this a bit smoother because metrics and parameters across experiments show up on one page instead of forcing you to hunt through separate runs, which matters when a reviewer wants to see how a model's performance changed across retraining cycles.
Databricks Jobs for orchestration. The Kedro pipeline gets triggered as tasks inside a Databricks Job rather than a cron entry calling kedro run on a VM somewhere. Lakeflow Jobs handles the scheduling, retries, and alerting, and because it's tied to the same workspace as Unity Catalog and MLflow, the lineage from raw ingestion through to a registered model is visible in one place instead of stitched together from logs.
What we didn't solve on day one
I want to be honest that this wasn't a clean, one sprint migration. Click based exit codes from the packaged Kedro entrypoint confused job status reporting for a few weeks until we adjusted how failures were surfaced. Some non-Spark datasets needed path rewrites we didn't catch until a run failed on a missing local folder that used to exist in the old VM setup. And cloning this project structure for a second requirement is still a manual exercise, we're building an internal template for that now rather than trusting copy paste inside the workspace to preserve config isolation between clients.
The part worth remembering
None of this was about choosing Databricks over Kedro or the other way around. It was about letting each tool do the part it's actually good at. Kedro still owns the pipeline logic and the testing discipline that got this project through validation in the first place. Databricks owns the parts that used to be the actual pain, which cluster to use, who can see which column, and where a model's lineage actually lives. For a project handling patient data, that second half turned out to matter a lot more than we expected going in.
References
Salman Ahmed
