cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Automating Migration of Delta Live Tables Pipelines Across Environments Using Azure DevOps CI/CD

Kumarn031425
New Contributor

I am seeking guidance on automating the migration of Delta Live Tables (DLT) pipelines across various environments—specifically from development to testing, and ultimately to production—utilizing Azure DevOps for Continuous Integration and Continuous Deployment (CI/CD).

Current Setup:

  • Version Control: Our notebooks and configurations, is stored in Azure Repos.
  • CI/CD Tool: We are using Azure DevOps to manage our CI/CD processes.
  • Databricks Integration: Integration between Azure DevOps and our Databricks workspace has been established.

Objective:

To establish an automated, reliable, and efficient process for migrating DLT pipelines across environments using Azure DevOps CI/CD pipelines.

Specific Questions:

  1. Best Practices: What are the recommended best practices for automating the migration of DLT pipelines using Azure DevOps CI/CD?
  2. Process Flow: Could you provide a detailed process flow or pipeline structure that facilitates this automation?
  3. Resources: Are there any tutorials, documentation, or community posts that offer step-by-step guidance on implementing this automation?

I appreciate any insights, experiences, or resources the community can share to assist in achieving this automation.

Thank you in advance for your support.

Best regards,

Kumar

1 REPLY 1

ashraf1395
Valued Contributor III

Hi there @Kumarn031425 , 
I guess, This video tutorial will answer most of your questions : https://youtu.be/SZM49lGovTg?si=X7Cwp0Wfqlo1OnuS

Here , deployment of workspace resources using databricks azure devops and databeicks asset bundles tutorial is given which are best recommended tools to handle migration of deployment of objects in databricks.

 

 

Many organizations choose Azure DevOps for automated deployments on Azure. When deploying to Databricks you can take similar deploy pipeline code that you use for other projects but use it with Databricks Asset Bundles. This video shows most of the steps involved in setting this up by following ...

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group