For many data scientists with a more statistical background or business-facing data professionals, MLOps can seem like a boring or bureaucratic job.
Conversely, for companies, control over code processes and model releases is a crucial requirement.
Databricks is here to help!
With a strong suite of tools to enhance the development experience and to enable versioning and deployment of models at scale, Databricks’ offerings are the best in class for MLOps.
In this article, I want to help Data Scientists understand and get started with MLOps tools and processes that can help you scale your work to meet the needs of your business.
Below, I have outlined some of the key elements that we recommend for productionizing your models, along with a link to a repository where you can find out how these things can be used in practice!
1. Databricks Lakeflow Jobs allow you to orchestrate and automate data and ML workflows such as data preprocessing, training, running inferences, and deploying models with:
- Flexible Scheduling: CRON-based scheduling for Notebooks, Python and SQL scripts - YES, you can schedule Notebooks!
- Parameterizing: You can parameterize your jobs to make your notebooks point to your Development, Staging, or Production catalogs. This is extremely useful because it means you can use the same code, just with different parameters, across environments
- Programmatic Control: Manageable via Databricks API, CLI, or SDK for CI/CD integration
- Lower costs: You can spend up to 3x less if you train your models or run your inferences using Jobs Compute instead of Interactive Compute.
2. MLflow 3.0 enhances ML tracking, evaluation, and deployment for Classic ML and Gen AI. MLflow allows you to save and track models trained in any framework to a unified format. Key features include:
- Seamless Unity Catalog Integration: All model metrics, parameters, and traces are accessible from Unity Catalog for cross-workspace governance and lineage. Models are first-class objects in Unity Catalog, so ML professionals enjoy the same level of governance as they would get on a Delta table in UC. You can save and load your models to and from the UC in a user-friendly way. You can also use UC to monitor your model's lineage and permissions- You heard it correctly: you don't need to manage Pickle files for your models!!
- Lifecycle Management: The UC allows you to control model versions, assign aliases such as "champion” to your best model version and "challenger” to the latest promising model version, and even promote model versions from one environment to another.
- Enhanced Experiment Tracking: Centralized tracking and analysis of models and agents with improved visibility into metrics directly from the Unity Catalog.
- Deployment Jobs: Automated evaluation, approval, and deployment of new model versions, including human-in-the-loop approvals and audit trails.
3. Databricks Asset Bundles (DABs) allow you to apply infrastructure-as-code principles to your jobs and notebooks:
- Template-Driven Deployments: You can use YAML templates for versioned, reusable configurations - This means you can, for example, create a workflow from the UI, export its DAB template, and then use it to redeploy it across environments (dev, staging, prod), without having to recreate it manually!
- Environment Isolation: Deploy to different environments with specific overrides.
- CI/CD Integration: Supports source control, code review, automated testing, and continuous deployment.
- Multiple options for execution: You can create/modify resources using DABs by executing commands from the Databricks Command Line Interface (CLI), from the Databricks UI, or even using Python code!
Want to see it in practice? Check out this Repository for an end-to-end Databricks MLOps example. In it, you will implement a pipeline that features:
- A job for ingesting mock data into a feature table
- A job for training a classification model and storing it in the Unity Catalog
- A job that uses the trained model to run inferences on top of newly collected samples (batch inference)
- An MLflow 3.0 deployment job for automating the process of defining the model used for running the production inferences
The repository is structured in such a way that there are 2 branches, and each one is connected to a different development environment (Dev and Prod). Each branch is associated with a deployment task in GitHub Actions, so that in case someone pushes something to the branches, the code changes are automatically propagated to the Databricks Workspace. There is also an Azure DevOps pipeline file for reference, in case you want to implement something similar in it.
Please make sure to check it out and run it on your end, and if you have any questions, do not hesitate to reach out!
