Best Practices for CI/CD on Databricks
For CI/CD and software engineering best practices with Databricks notebooks we recommend checking out this best practices guide (AWS, Azure, GCP).
For CI/CD and local development using an IDE, we recommend dbx, a Databricks Labs project which offers an extension to the Databricks CLI that allows you to develop code locally and then submit against Databricks interactive and job compute clusters from your favorite local IDE (AWS, Azure, GCP).
For MLOps specific guidance, we recommend checking out the Big Book of MLOps.
For infrastructure as code to deploy and configure Databricks workspaces, access control, and security in an automated and scalable way, we recommend our full-featured and fully supported Databricks Terraform Provider.
For further information and a deep dive on these topics and more, we recommend getting in touch with your Databricks account team.
Have you implemented a CI/CD pipeline with Databricks? What tools did you use? Let us know how it went in the comments!
