https://community.databricks.com/t5/technical-blog/ci-cd-integration-with-databricks-workflows/ba-p/81821 <DIV class="page" title="Page 1"> <DIV class="section"> <DIV class="layoutArea"> <DIV class="column"> <P><FONT size="5"><U><STRONG>CICD with Databricks Asset Bundles, Workflows and Azure DevOps </STRONG></U></FONT></P> <P><SPAN>In this article you will learn how to set up Databricks Workflows with CI/CD. There are two essential components needed for a complete CI/CD setup of Workflow jobs. </SPAN></P> <UL> <LI> <P><SPAN>Databricks Asset Bundles (DABs): </SPAN><SPAN><A href="https://learn.microsoft.com/en-us/azure/databricks/dev-tools/bundles/" target="_blank" rel="noopener">https://learn.microsoft.com/en-us/azure/databricks/dev-tools/bundles/</A> </SPAN></P> </LI> <LI> <P><SPAN>AzureDevOps pipeline.</SPAN></P> </LI> </UL> <P><U><STRONG><SPAN>Getting started with Databricks Asset Bundles</SPAN></STRONG></U></P> <P><SPAN>We can use Databricks Asset Bundle(DABs) with the Databricks CLI from any terminal to deploy Workflows. Please note that Databricks Asset Bundles (DABs) are available in the latest version of databricks-cli (v0.205.0 and above). The legacy version will not work. </SPAN></P> <UL> <LI><SPAN>Installation of Databricks-cli in local terminal:</SPAN> <UL> <LI><SPAN>Run the below commands to install databricks-cli in the local terminal:</SPAN></LI> <LI><SPAN>For more details, follow the below link to install databrick cli(v0.205 and&nbsp; above): <A href="https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/install#--curl-installation-for-linux-macos-and-windows" target="_blank" rel="noopener">https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/install#--curl-installation-for-linux-macos-and-windows</A></SPAN><LI-CODE lang="python">curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh</LI-CODE></LI> </UL> </LI> </UL> <UL> <LI><SPAN>Authenticate your databricks cli to dev workspace(Using PAT):</SPAN> <UL> <LI><SPAN>Follow the steps from the below link to authenticate local terminal to dev workspace using personal access token PAT: </SPAN><A href="https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/authentication#--azure-databricks-personal-access-token-authentication" target="_blank" rel="noopener"><SPAN>https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/authentication#--azure-databricks-personal-access-token-authentication</SPAN></A></LI> </UL> </LI> </UL> </DIV> </DIV> </DIV> </DIV> <DIV class="page" title="Page 2"> <DIV class="section"> <DIV class="layoutArea"> <DIV class="column"> <UL> <LI><SPAN>Authenticate your databricks cli to dev workspace(Using Service principal):</SPAN> <UL> <LI><SPAN>Follow the steps from the below link to authenticate local terminal to dev workspace using a Service principal: <A href="https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/authentication#--microsoft-entra-id-service-principal-authentication" target="_blank" rel="noopener">https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/authentication#--microsoft-entra-id-service-principal-authentication</A> </SPAN></LI> </UL> </LI> </UL> <P><SPAN>From your local terminal, run the following command:</SPAN></P> <LI-CODE lang="python">databricks bundle init</LI-CODE> <P><SPAN>When prompted, select Python project and provide a project name (e.g., </SPAN><I><SPAN>demo_wf</SPAN></I><SPAN>). </SPAN><SPAN>After you complete the prompts, a folder will be generated with the project name containing all the components needed for a Workflow, as shown below.</SPAN></P> <span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="SashankKotta_1-1723049093943.png" style="width: 624px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/10207iA7C2D76E849C4477/image-dimensions/624x63?v=v2" width="624" height="63" role="button" title="SashankKotta_1-1723049093943.png" alt="SashankKotta_1-1723049093943.png" /></span> <P>&nbsp;</P> </DIV> <DIV class="column"><SPAN>Next, navigate to the project directory.</SPAN><LI-CODE lang="python">cd demo_wf</LI-CODE> <P><SPAN>We need the notebooks in .ipynb format inside the </SPAN><SPAN>src </SPAN><SPAN>folder. These notebook files will be the respective tasks in the Workflow. We can also create DLT pipelines and libraries as individual tasks. </SPAN></P> <P><SPAN>Inside the </SPAN><SPAN>resources </SPAN><SPAN>folder, we will have a YAML file called </SPAN><SPAN>&lt;project_name&gt;_job.yml </SPAN><SPAN>(in our case, </SPAN><SPAN>demo_wf_job.yml</SPAN><SPAN>). This file defines the task flow:</SPAN></P> <LI-CODE lang="python">tasks: - task_key: task1 job_cluster_key: job_cluster notebook_task: notebook_path: ../src/notebook_1.ipynb - task_key: task2 job_cluster_key: job_cluster notebook_task: notebook_path: ../src/notebook_2.ipynb depends_on: - task_key: task1</LI-CODE></DIV> </DIV> </DIV> </DIV> <DIV class="page" title="Page 3"> <DIV class="section"> <DIV class="layoutArea"> <DIV class="column"> <P><SPAN>After navigating to the project directory (</SPAN><SPAN>demo_wf</SPAN><SPAN>), run the following command to catch any syntax errors prior to deployment.</SPAN></P> <LI-CODE lang="python">databricks bundle validate</LI-CODE> <P><SPAN>Finally, run the command to deploy the Workflow in development mode.</SPAN></P> <LI-CODE lang="python">databricks bundle deploy -t dev</LI-CODE> <P><SPAN>At this point, you have a sample project with a Workflow deployed to your Databricks Workspace. </SPAN><SPAN>The same commands can be run from a build pipeline in Azure DevOps, and that will complete the CICD setup. </SPAN></P> <P><U><STRONG>Using DABs in Azure DevOps pipelines:</STRONG></U></P> <P><SPAN>To begin, we need an Azure virtual machine to run commands as an agent for our DevOps pipeline. Create a virtual machine in Azure, assign a Network security group, and set inbound rules to allow SSH (port 22) from your IP address so you can&nbsp;</SPAN><SPAN>connect using SSH and do the setup on this virtual machine. While creating the VM, we will be asked to download a .pem file - keep it safe as it is needed while connecting to the VM through an SSH.</SPAN></P> </DIV> </DIV> </DIV> </DIV> <DIV class="page" title="Page 4"> <DIV class="section"> <DIV class="layoutArea"> <DIV class="column"> <P><SPAN>The next step is to install the databricks-cli on this VM and configure this machine as an agent for your Azure agent pool. If you have setup the inbound networking rules correctly, you can connect to the VM using the command:</SPAN></P> <LI-CODE lang="python">ssh -i &lt;path_to_pem&gt;/&lt;file_name&gt;.pem &lt;username&gt;@&lt;hostname&gt;</LI-CODE> <P><SPAN>Now install the databricks-cli:</SPAN></P> <LI-CODE lang="python">curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sudo sh</LI-CODE> <P><SPAN>If you get any errors concerning unzip, then please install unzip using the below commands and re-run the above curl command:</SPAN></P> <LI-CODE lang="python">sudo apt update -y sudo apt install unzip -y</LI-CODE> <P><SPAN>Next, configure the VM to run as an agent in the Azure agent pool by following the steps below.</SPAN></P> <OL> <LI><STRONG style="font-family: inherit;">Create a Self-hosted Agent in Azure DevOps:&nbsp;</STRONG>To create a self-hosted agent, go to <STRONG style="color: #1b3139; font-family: inherit;">Project Settings</STRONG><SPAN> (Bottom left) and select the </SPAN><STRONG style="color: #1b3139; font-family: inherit;">Agent pools</STRONG><SPAN> option under the </SPAN><STRONG style="color: #1b3139; font-family: inherit;">Pipelines</STRONG><SPAN> section. Press the </SPAN><STRONG style="color: #1b3139; font-family: inherit;">Add pool</STRONG><SPAN> button and configure the agent:</SPAN></LI> </OL> </DIV> </DIV> </DIV> </DIV> <DIV class="page" title="Page 5"> <DIV class="section"> <DIV class="layoutArea"> <DIV class="column"> <OL> <LI style="list-style-type: none;"> <OL> <UL> <LI style="font-weight: 400;" aria-level="2"><SPAN>Select the </SPAN><STRONG>Pool type</STRONG><SPAN> as Self-hosted</SPAN></LI> <LI style="font-weight: 400;" aria-level="2"><SPAN>Add a descriptive pool </SPAN><STRONG>Name</STRONG><SPAN> (in this example </SPAN><SPAN>my-demo-pool</SPAN><SPAN>)</SPAN></LI> <LI style="font-weight: 400;" aria-level="2"><SPAN>Check the </SPAN><STRONG>Pipeline permissions</STRONG><SPAN> box so you do not need to grant permission manually</SPAN></LI> <LI style="font-weight: 400;" aria-level="2">Click on the <STRONG style="color: #1b3139; font-family: inherit;">Create</STRONG><SPAN> button</SPAN></LI> <LI style="font-weight: 400;" aria-level="2">For more details, refer to the link: <A style="font-family: inherit; background-color: #ffffff;" href="https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/pools-queues?view=azure-devops&amp;tabs=yaml%2Cbrowser" target="_blank" rel="noopener"><SPAN>Create or manage agent pool</SPAN></A></LI> </UL> </OL> </LI> <LI style="font-weight: 400;" aria-level="2"><SPAN>On the top-right corner, press the </SPAN><SPAN>New Agent </SPAN><SPAN>button. You can create&nbsp;</SPAN>Windows, macOS, and Linux agents. Based on your VM, select the appropriate OS, then follow the instructions; in our case, we used an Ubuntu image so that it will be a Linux agent. <UL> <LI style="font-weight: 400;" aria-level="2"><SPAN>Connect to the VM using the SSH</SPAN></LI> <LI style="font-weight: 400;" aria-level="2">Download the agent then extract it to a folder using the below linux commands in the VM terminal.<LI-CODE lang="python">#Create a directory named myagent mkdir myagent #Navigate to that agent. cd myagent #Download the linux agent zip from the link given in the instructions using linux wget https://vstsagentpackage.azureedge.net/agent/3.236.1/vsts-agent-linux-x64-3.236.1.tar.gz #Unzip the agent file from the downloaded zip tar zxvf ~/Downloads/vsts-agent-linux-x64-3.236.1.tar.gz </LI-CODE></LI> </UL> </LI> <LI><SPAN><SPAN><SPAN>Configure the agent by running the config script in the VM terminal:</SPAN></SPAN></SPAN><LI-CODE lang="python">./config.sh</LI-CODE> <UL> <LI><SPAN>Fill in the following prompts: </SPAN> <UL> <LI><SPAN>Server URL: Copy and paste the organization URL, which looks like the </SPAN><SPAN>following: </SPAN><SPAN><A href="https://dev.azure.com/" target="_blank" rel="noopener">https://dev.azure.com/</A>&lt;my-organization-name&gt;</SPAN></LI> <LI><SPAN>Personal Access Token (PAT): Go to the Personal Access Tokens option&nbsp;</SPAN><SPAN>under the User Settings icon. Ensure you generate a PAT with Read &amp;&nbsp;</SPAN><SPAN>manage access to the Agent pools.</SPAN></LI> <LI><SPAN>Agent pool name: The newly created pool, which is the my-demo-pool </SPAN><SPAN>in our case </SPAN></LI> <LI><SPAN>Agent Name: Give a meaningful name or stay with the default </SPAN></LI> <LI><SPAN>Work folder: Press enter for the default </SPAN></LI> <LI><SPAN>Agent as Service: Press enter to use the default.</SPAN></LI> </UL> </LI> </UL> </LI> <LI><SPAN>Run the agent by executing the run script. </SPAN> <UL> <LI><LI-CODE lang="python">./run.sh</LI-CODE></LI> <LI><SPAN>Once done, you can see that the Agent is up and running under the Agents panel. The self-hosted agent is connected to Azure DevOps and listens for new jobs. </SPAN></LI> <LI><SPAN>For more details, refer to the link: </SPAN><A href="https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/linux-agent?view=azure-devops" target="_blank" rel="noopener"><SPAN>Self-hosted agent</SPAN></A></LI> </UL> </LI> <LI><SPAN>Now that the agent VM is configured, the next step is to create azure-pipelines.yml (DevOps </SPAN><SPAN>pipeline). Yaml should look like the one below. </SPAN></LI> </OL> </DIV> </DIV> </DIV> </DIV> <DIV class="page" title="Page 6"> <DIV class="section"> <DIV class="layoutArea"> <DIV class="column">&nbsp;</DIV> </DIV> </DIV> </DIV> <DIV class="page" title="Page 7"> <DIV class="section"> <DIV class="layoutArea"> <DIV class="column"><LI-CODE lang="python"># Starter pipeline # Start with a minimal pipeline that you can customize to build and deploy your code. # Add steps that build, run tests, deploy, and more: # https://aka.ms/yaml trigger: - main pool: my-demo-pool steps: - script: echo "Hello, world!" displayName: 'Run a one-line script' - script: | echo Add other tasks to build, test, and deploy your project. echo See https://aka.ms/yaml displayName: 'Run a multi-line script' - task: Bash@3 inputs: targetType: 'inline' script: | # Write your commands here echo 'Hello world' touch ~/.databrickscfg echo "[DEFAULT]" &gt; ~/.databrickscfg echo "host = &lt;workspace_host_url&gt;" &gt;&gt; ~/.databrickscfg echo "azure_workspace_resource_id = &lt;Azure_sp_resource_id&gt;" &gt;&gt; ~/.databrickscfg echo "azure_tenant_id = &lt;tenant_id&gt;" &gt;&gt; ~/.databrickscfg echo "azure_client_id = &lt;spn_client_id&gt;" &gt;&gt; ~/.databrickscfg echo "azure_client_secret =&lt;client_secret&gt;" &gt;&gt; ~/.databrickscfg cat ~/.databrickscfg databricks bundle validate databricks bundle deploy -t dev </LI-CODE> <P><SPAN>T</SPAN><SPAN>o ensure our CI/CD is working as expected, the VM Agent should be up and running under the Agents panel (Project settings &gt; Agent pools &gt; Agents tab). The folder structure in the main branch of Azure DevOps should look this:</SPAN></P> <span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="SashankKotta_0-1723049447830.png" style="width: 656px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/10209iA61E059FCE902E61/image-dimensions/656x328?v=v2" width="656" height="328" role="button" title="SashankKotta_0-1723049447830.png" alt="SashankKotta_0-1723049447830.png" /></span></DIV> <DIV class="column">&nbsp;</DIV> <DIV class="column"><SPAN>Any changes to the Azure DevOps main branch should be deployed/reflected in the Workflow jobs of your Databricks Workspace.</SPAN></DIV> </DIV> </DIV> </DIV> <DIV class="page" title="Page 8"> <DIV class="section"> <DIV class="layoutArea"> <DIV class="column"><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="SashankKotta_2-1723049195748.png" style="width: 657px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/10208iDD0FAFEA32A4C838/image-dimensions/657x269?v=v2" width="657" height="269" role="button" title="SashankKotta_2-1723049195748.png" alt="SashankKotta_2-1723049195748.png" /></span> <P>&nbsp;</P> </DIV> </DIV> <DIV class="layoutArea"> <DIV class="column"> <P><STRONG><U>Conclusion:</U></STRONG></P> <P><SPAN>In conclusion, setting up Databricks Workflows with CI/CD involves two key components: Databricks Asset Bundles (DABs) and an Azure DevOps pipeline. By using DABs with the Databricks CLI, you can easily deploy workflows from any terminal. Integrating this setup with Azure DevOps requires configuring a virtual machine as an agent, installing necessary tools, and creating a pipeline to automate deployments, ensuring seamless updates to your Databricks Workflows. </SPAN></P> </DIV> </DIV> </DIV> </DIV> <DIV class="page" title="Page 9">&nbsp;</DIV> Tue, 27 Aug 2024 12:50:07 GMT SashankKotta 2024-08-27T12:50:07Z https://community.databricks.com/t5/technical-blog/ci-cd-integration-with-databricks-workflows/ba-p/81821 <P>CI/CD Integration with Databricks Workflows: A Practical Guide: This article provides a hands-on approach to setting up Databricks Workflows with CI/CD. It focuses on the use of Databricks Asset Bundles (DABs) and Azure DevOps pipeline, detailing the process of deploying workflows from any terminal and creating a pipeline for automated deployments.</P> Tue, 27 Aug 2024 12:50:07 GMT https://community.databricks.com/t5/technical-blog/ci-cd-integration-with-databricks-workflows/ba-p/81821 SashankKotta 2024-08-27T12:50:07Z https://community.databricks.com/t5/technical-blog/ci-cd-integration-with-databricks-workflows/bc-p/84403#M256 <P><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/108479">@SashankKotta</a>&nbsp; Awesome, thank you for sharing this !!!</P> Tue, 27 Aug 2024 13:37:14 GMT https://community.databricks.com/t5/technical-blog/ci-cd-integration-with-databricks-workflows/bc-p/84403#M256 Vishwajeet_Pol 2024-08-27T13:37:14Z https://community.databricks.com/t5/technical-blog/ci-cd-integration-with-databricks-workflows/bc-p/89079#M266 <P><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/108479">@SashankKotta</a>&nbsp;Thanks for sharing&nbsp;</P> Sun, 08 Sep 2024 15:53:01 GMT https://community.databricks.com/t5/technical-blog/ci-cd-integration-with-databricks-workflows/bc-p/89079#M266 Ajay-Pandey 2024-09-08T15:53:01Z https://community.databricks.com/t5/technical-blog/ci-cd-integration-with-databricks-workflows/bc-p/95393#M324 <P>Thank for sharing the helpful instruction for ci/cd with azure databricks&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/108479">@SashankKotta</a>&nbsp;</P> Tue, 22 Oct 2024 02:16:07 GMT https://community.databricks.com/t5/technical-blog/ci-cd-integration-with-databricks-workflows/bc-p/95393#M324 prachy 2024-10-22T02:16:07Z https://community.databricks.com/t5/technical-blog/ci-cd-integration-with-databricks-workflows/bc-p/96678#M341 <P><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/108479">@SashankKotta</a>&nbsp;Thank you for sharing this. just wondering when we are deploying Databricks with standard deployment which has 2 VNets (VNet Injection) If we already have another VNet for&nbsp;<SPAN>AzureDevOps pipeline, can we do VNet peering to Transit VNet to connect to compute workspace or any suggestions?&nbsp;<BR />ps. I am trying to do not creating new VM in Transit VNet to use the existing if it is possible.</SPAN></P><P><SPAN>Thank you</SPAN></P> Tue, 29 Oct 2024 16:11:45 GMT https://community.databricks.com/t5/technical-blog/ci-cd-integration-with-databricks-workflows/bc-p/96678#M341 prachy 2024-10-29T16:11:45Z https://community.databricks.com/t5/technical-blog/ci-cd-integration-with-databricks-workflows/bc-p/96756#M343 <P>How do I compose the DAB YAML ?&nbsp; How do I know about the keys for the key-value pair ?&nbsp; &nbsp;Thanks</P> Wed, 30 Oct 2024 06:57:26 GMT https://community.databricks.com/t5/technical-blog/ci-cd-integration-with-databricks-workflows/bc-p/96756#M343 d_kabilan 2024-10-30T06:57:26Z