Most teams I work with don't struggle with the AI part of MLOps. They struggle with the boring parts: getting code from a notebook into production without breaking it, knowing when a model has gone stale, and giving humans a clean way to make and approve changes.
The good news is that Databricks offers a range of resources to build credible end-to-end MLOps pipelines. The MLOps Quickstart repository (databricks-solutions/mlops-quickstart) is a solid starting point for a repository template you can follow. It walks through the complete Iris Model problem, where your goal is to classify flower species based on their measurements.
It's worth knowing what's actually in the repo before you clone it. The quickstart is organized around the same asset types you'll maintain in any real MLOps project:
- Notebooks for the ML work: data preprocessing, model training, and inference.
- Infrastructure-as-code resources: the Declarative Automation Bundle's main file (databricks.yml) and the job definitions it deploys (under the resources folder).
- Config files: pinned dependencies, the CI/CD workflow definitions, and environment parameters.
- Skills: a reusable .assistant/skills set you can plug into Genie Code to adapt the repo to your own use case (more on this below).
The project implements a complete pipeline with data preprocessing, training, deployment, and inference jobs, all orchestrated by Declarative Automation Bundles and wired into GitHub Actions or Azure DevOps for CI/CD. If you've never built this before, clone it, run through it, and use it as your skeleton.
Here is the implementation plan I'd suggest, in order:
- First, create a repository in your preferred Git repository tool, such as GitHub, Azure DevOps, Bitbucket, or GitLab. Clone this repository from Databricks and store all your notebooks and jobs from there.
- Second, build your feature preprocessing pipeline to ensure data consistency. To train a good model, you need good data. Start with a notebook for creating a good feature table on top of which you'll train your model, then automate this notebook through Lakeflow Jobs.
- Third, create a notebook to train and register your model artifacts in the Unity Catalog, and adopt the Challenger/Champion alias pattern - every training produces a Challenger, selected based on the best run of your experiment; and, once validated, your models become Champions. Saving your models to the UC allows for you to reuse your trained models while preserving lineage and traceability!
- Fourth, define your pipeline as code with a databricks.yml Asset Bundle, parameterized per environment (dev, stg, prod), so a single databricks bundle deploy --target <dev, stg, prod> deploys the whole stack.
- Fifth, wire CI/CD: GitHub Actions or Azure DevOps to watch the dev/main branches and trigger the right bundle deploy on push.
- Sixth, separate training from deployment using MLflow 3 Deployment Jobs, which automate evaluation and deployment whenever a new model version is registered in Unity Catalog.
- These jobs simplify the setup of model pipelines by incorporating human-in-the-loop approvals and audit trails. This oversight is particularly important for sensitive industries like healthcare to ensure senior-level checks and quality control before live deployment.
- Finally, set up batch inference or serving endpoints against the Champion alias and persist predictions to an Inference Table for downstream consumption and monitoring. This is one of the core aspects of your Data Science projects: to decide whether you will have to run your predictions on a schedule (more aligned to batch inference) or in near real-time (aligned to serving endpoints, where you will have a dedicated machine running your model and allowing you to treat it as an API). This will ultimately depend on your business needs, latency requirements, request volume, and cost tolerance.
Additionally, you will see that the mlops-quickstart project also features an .assistant/skills folder. This is a set of skills you can use to adapt this repository for your projects:
You can add these skills to your Genie Code settings by following this guide: https://docs.databricks.com/aws/en/genie-code/skills and it will supercharge your Genie Code abilities with MLOps skills, and then ask it to adapt the repository to:
- Ingest and preprocess your own data.
- Train the model you want to train with the framework you specify.
- Set up jobs for your MLOps project.
- .. and more! All tailored to your specific needs.
Keep in mind that MLOps is the practice of managing three assets together across their full lifecycle: your data, your models, and your code. This is what makes "production-ready" concrete. With this lens, before you call any MLOps setup "done", run through this checklist. If you can't tick every box, you have a gap worth closing before the project goes live:
Code lives in Git, not in shared notebooks in the workspace, with branch protection on main. Each developer should clone the repository under their personal folder in the workspace to develop and ensure they push their changes to remote branch so that everything gets properly tracked and audited.
A databricks.yml Asset Bundle deploys all jobs and resources, parameterized per environment, and a CI/CD pipeline runs databricks bundle deploy on the right branch events.
Models are registered in Unity Catalog with Challenger/Champion aliases.
An MLflow 3 Deployment Job gates promotion with evaluation and human approval.
requirements.txt with exact pinned versions, logged to MLflow alongside the model.
Inference results land in an Inference Table for traceability.
Data Quality Monitoring is enabled on feature tables and the inference table (InferenceLog).
Alerting is configured on monitoring metrics and on job failures, routed to the right team.
Service principals (not personal tokens) authenticate CI/CD, with secrets in place for sensitive information.
Permissions on jobs, models, and tables differ between dev and prod.
Unit tests on transformations and an integration test on the full pipeline run in CI.
A rollback path exists — promoting a previous model version takes just some small commands, not a complex rebuild, for example: alias reassignment or bundle redeploy.
Real projects can outgrow the quickstart fast, and there are a few patterns I'd re-emphasize and add as the engagement matures:
- Always ship a requirements.txt (or pyproject.toml) and pin library versions exactly - scikit-learn==1.4.2, not scikit-learn>=1.4 - because a silent minor-version bump in pandas or scikit-learn is one of the most common ways a working model suddenly produces different predictions in prod. Log the same pinned environment to MLflow at training time so the serving environment is reproducible.
- Consider wrapping your "predict" method around a Spark UDF for distributed inferences in large-scale datasets
- Enable Data Quality Monitoring on both your feature tables and your inference table - InferenceLog profile type for the latter - to get drift, data quality, and model performance metrics out of the box, plus an auto-generated dashboard.
- Add unit tests for transformations and integration tests for the end-to-end pipeline; the Databricks Academy "Testing Strategies with Databricks" module is a good reference.
- Promote models across environments instead of retraining per environment when training cost or determinism is a concern; the video "From Notebook to Production: MLOps Quickstart" provides a detailed walkthrough on it.
- Check out Genie ZeroOps (currently in Private Preview) to help you monitor and maintain your ML runs and operations.
- Finally, default to serverless or job compute over interactive clusters - it's materially cheaper and easy to set up using bundles.
Start with the quickstart, then layer these in as you grow. The Big Book of MLOps is a deeper read once you're past the basics.
