https://community.databricks.com/t5/technical-blog/azure-devops-with-databricks-dabs-step-by-step-integration-guide/ba-p/141451 <P><SPAN>Establishing a trusted </SPAN><STRONG><I>Continuous Integration/Continuous Deployment (CI/CD)</I></STRONG> <SPAN>process is crucial for effectively managing the lifecycle of your data and AI workloads in Azure Databricks. However, with numerous setup options and a constantly evolving platform, the big question is, "Where do I even start?" This comprehensive blog cuts through the complexity to outline a </SPAN><A href="https://docs.databricks.com/aws/en/dev-tools/ci-cd/best-practices" target="_blank" rel="noopener"><STRONG>recommended CI/CD approach</STRONG></A> <SPAN>for Azure Databricks.&nbsp;</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="jim_thorstad_0-1776186435312.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/25999i080FF82107D99CC4/image-size/medium?v=v2&amp;px=400" role="button" title="jim_thorstad_0-1776186435312.png" alt="jim_thorstad_0-1776186435312.png" /></span></P> <P><SPAN>We focus on the latest best practices that support strong governance with minimal administrative overhead, even when </SPAN><STRONG>supporting</STRONG> <STRONG>multiple teams in a shared infrastructure</STRONG><SPAN>. It's tailored for developers who enjoy working in the Databricks workspace UI, though it also fully supports IDEs like Visual Studio Code.</SPAN></P> <P><SPAN>The recommended approach uses </SPAN><A href="https://docs.databricks.com/aws/en/dev-tools/bundles/" target="_blank" rel="noopener"><STRONG>Declarative Automation Bundles</STRONG></A> <SPAN>(DABs; formerly known as Databricks Asset Bundles) for deployment. Bundles seamlessly integrate with </SPAN><A href="https://docs.databricks.com/aws/en/repos/" target="_blank" rel="noopener"><SPAN>Databricks Git Folders</SPAN></A><SPAN> for version control, Azure DevOps for automation, and Azure DevOps service connections for Databricks connectivity. At the end of the blog, I compare the DevOps service connection and Databricks OIDC approaches for integrating DevOps and Databricks for your reference.</SPAN></P> <P><SPAN>Finally, we’ll follow a </SPAN><A href="https://docs.databricks.com/aws/en/dev-tools/ci-cd/best-practices#branching-strategy" target="_blank" rel="noopener"><SPAN>simple but effective trunk-based branching strategy</SPAN></A><SPAN> in DevOps.&nbsp; Staging will be deployed automatically after feature branch pull requests are approved and merged into main. Production deployment from main will be contingent on a successful staging deployment and will require an additional approval step, which we will implement.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_1-1776186629097.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26001iF57EF441D810B508/image-size/large?v=v2&amp;px=999" role="button" title="jim_thorstad_1-1776186629097.png" alt="jim_thorstad_1-1776186629097.png" /></span></P> <P><SPAN>This blog covers a wide range of topics, but by guiding you toward the optimal choices, it will save you significant time and effort in the long run. The content is divided into three parts:</SPAN></P> <OL> <LI style="font-weight: 400;" aria-level="1"><SPAN>Aligning your infrastructure and governance requirements</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Setting up a versioned bundle development process</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Setting up an Azure DevOps pipeline for deployment to staging and production</SPAN></LI> </OL> <H1><SPAN>Part 1: Align Your Infrastructure and Governance Requirements</SPAN></H1> <P><SPAN>Every organization needs to decide what level of separation and organization is required for Databricks workspaces, deployment environments, Unity Catalog, DevOps projects, and Git repositories. There is no one right answer for every organization, and we can only touch on the considerations before moving on to the article's topic.&nbsp;</SPAN></P> <TABLE> <TBODY> <TR> <TD width="121px" height="50px"> <P><STRONG>Topic</STRONG></P> </TD> <TD width="355px" height="50px"> <P><STRONG>Consideration</STRONG></P> </TD> <TD width="344px" height="50px"> <P><STRONG>For This Article</STRONG></P> </TD> </TR> <TR> <TD width="121px" height="77px"> <P><SPAN>Environments</SPAN></P> </TD> <TD width="355px" height="77px"> <P><SPAN>Three is common, but some may use two, and others four or more.</SPAN></P> </TD> <TD width="344px" height="77px"> <P><SPAN>We’ll use: </SPAN><STRONG>dev, staging, and prod</STRONG></P> </TD> </TR> <TR> <TD width="121px" height="289px"> <P><SPAN>Databricks Workspaces</SPAN></P> </TD> <TD width="355px" height="289px"> <P><SPAN>Generally, workspaces align with environments. Larger organizations may also organize business units in separate workspaces.</SPAN></P> <BR /> <P><SPAN>An advantage of three workspaces is that the workspace admin can freely enable </SPAN><STRONG>preview features</STRONG><SPAN> and other settings in the “dev” workspace and be certain not to affect the staging and prod workloads.</SPAN></P> </TD> <TD width="344px" height="289px"> <P><SPAN>We’ll show how multiple independent teams can use two workspaces (</SPAN><FONT face="andale mono,times" color="#3366FF">bu1-dev</FONT><SPAN> and </SPAN><FONT face="andale mono,times" color="#3366FF">bu1-prod</FONT><SPAN>) with reasonable governance. Staging will deploy to a secure folder in the dev workspace.</SPAN></P> <BR /> <P><SPAN>You can follow along and use 3 workspaces with dev, staging, and prod, each in its own workspace.</SPAN></P> </TD> </TR> <TR> <TD width="121px" height="132px"> <P><SPAN>Catalogs</SPAN></P> </TD> <TD width="355px" height="132px"> <P><SPAN>There are many strategies for organizing your data in Unity Catalog.</SPAN></P> </TD> <TD width="344px" height="132px"> <P><SPAN>We’ll name our catalogs </SPAN><FONT face="andale mono,times" color="#3366FF">bu1_dev</FONT>, <FONT face="andale mono,times" color="#3366FF">bu1_stg</FONT>, <SPAN>and&nbsp;</SPAN><FONT face="andale mono,times" color="#3366FF">bu1_prd</FONT>,<SPAN> demonstrating how independent business units can operate within a single Unity Catalog metastore.</SPAN></P> </TD> </TR> <TR> <TD width="121px" height="160px"> <P><SPAN>Azure DevOps Projects</SPAN></P> </TD> <TD width="355px" height="160px"> <P><SPAN>Smaller organizations use a single project to simplify setup, while larger organizations may require multiple projects to have hard boundaries between code repositories and approvers.</SPAN></P> </TD> <TD width="344px" height="160px"> <P><SPAN>We’ll use a project called </SPAN><FONT face="andale mono,times" color="#3366FF">bu1-team1</FONT>,<SPAN> allowing for independent teams (e.g., HR, Legal, Finance) within a “corporate” BU to operate securely within the shared workspaces alongside other BU teams.</SPAN></P> </TD> </TR> <TR> <TD width="121px" height="105px"> <P><SPAN>Git Repository</SPAN></P> </TD> <TD width="355px" height="105px"> <P><SPAN>There are pros and cons for a single “mono repo” versus multiple team or domain-based repos.</SPAN></P> </TD> <TD width="344px" height="105px"> <P><SPAN>Since we have already selected a team-based DevOps project, we’ll use its single </SPAN><STRONG>default repo</STRONG><SPAN>.</SPAN></P> </TD> </TR> </TBODY> </TABLE> <P><SPAN>Note: I encourage you to follow the example and organizational strategy outlined in this blog to see how the whole process works together effectively. Then you can consider what you learned, how your organization’s governance and organization needs may differ, and make the necessary changes. Having made these choices, let’s look over our organizational strategy visually.&nbsp;</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_2-1776186912117.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26003i24994D3FD7C4EF98/image-size/large?v=v2&amp;px=999" role="button" title="jim_thorstad_2-1776186912117.png" alt="jim_thorstad_2-1776186912117.png" /></span></P> <P><SPAN>Within our “corporate” organization (BU1), Team 2 handles HR data, and everything else is handled by Team 1. These two teams will work independently in our shared <FONT face="andale mono,times" color="#3366FF">bu1-dev</FONT> and <FONT face="andale mono,times" color="#3366FF">bu1-prod</FONT> workspaces. For this to work well, it is important to note that one or more individuals must be trusted by both teams to serve as workspace Administrators. If this can be arranged, infrastructure management is simplified; if not, teams 1 and 2 should be separated into their own dev/prod workspaces.</SPAN></P> <P><SPAN>Down the road, another division (BU2) could be acquired that shares the same Entra ID directory but elects to have separate workspaces and DevOps infrastructure. That’s okay, and it's the reason we’ll prefix our catalog names in Unity Catalog (e.g., <FONT face="andale mono,times" color="#3366FF">bu1_dev</FONT>, <FONT face="andale mono,times" color="#3366FF">bu2_dev</FONT>, …).</SPAN></P> <P><SPAN>We’ll align with the “mono repo” approach within each DevOps project and further organize our code using Declarative Automation Bundles. In this blog, we’ll use a single “Hello World” bundle to illustrate how the recommended development and CI/CD processes work. In practice, you will have many bundles for a given data domain (e.g., finance), and these can all be versioned in the same repo. We’ll use a single DevOps pipeline for staging and production deployments, but you may choose a different pipeline strategy.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_3-1776186945297.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26004iFC07BFFB27376B4A/image-size/large?v=v2&amp;px=999" role="button" title="jim_thorstad_3-1776186945297.png" alt="jim_thorstad_3-1776186945297.png" /></span></P> <H1><SPAN>Part 2 - Setting Up a Versioned Bundle Development Process</SPAN></H1> <P><SPAN>Databricks bundles and Git folders work seamlessly together with Azure DevOps, and set you up for success in your CI/CD process. Let’s jump into it.</SPAN></P> <H2><SPAN>Initial Azure DevOps Set Up</SPAN></H2> <P><SPAN>This is pretty straightforward and involves creating the team’s Azure DevOps project and initializing the repo. You do not need to create any branches–we will use the built-in </SPAN><STRONG>main branch</STRONG><SPAN> and create our </SPAN><STRONG>feature branches</STRONG><SPAN> within Databricks using the built-in Git integration UI.&nbsp;</SPAN></P> <P><SPAN>Create your project, initialize the main branch, and then go to the repo and copy the repository URL to your clipboard for the next step.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_4-1776186982114.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26005i663A964FC68AEB0D/image-size/large?v=v2&amp;px=999" role="button" title="jim_thorstad_4-1776186982114.png" alt="jim_thorstad_4-1776186982114.png" /></span></P> <H2><SPAN>Create a Linked Account to Access the Repo</SPAN></H2> <P><SPAN>In the Databricks workspace, each developer creates their own </SPAN><STRONG>Azure DevOps (MSFT Entra) linked account credential</STRONG><SPAN>, allowing them to authenticate to the Git repository. Do this now in your “dev” workspace on the </SPAN><STRONG>Settings &gt; User &gt; Linked accounts</STRONG><SPAN> page. You do not need to do this in your prod workspace because there is no Git integration required there (DevOps will push the code to prod).</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_5-1776187016894.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26006iEA3F1BF3EC19242C/image-size/large?v=v2&amp;px=999" role="button" title="jim_thorstad_5-1776187016894.png" alt="jim_thorstad_5-1776187016894.png" /></span></P> <H2><SPAN>Create a Git Folder that Points to the Repo</SPAN></H2> <P><SPAN>In their </SPAN><STRONG>Home</STRONG><SPAN> directory, each developer creates their own </SPAN><STRONG>Git folder </STRONG><SPAN>that points to the team’s repo. This step is done by each developer only once. Click </SPAN><STRONG>Create &gt; Git folder</STRONG><SPAN>.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_6-1776187041430.png" style="width: 508px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26007i00214B7521DA6DE9/image-dimensions/508x159?v=v2" width="508" height="159" role="button" title="jim_thorstad_6-1776187041430.png" alt="jim_thorstad_6-1776187041430.png" /></span></P> <P><SPAN>Paste in the repo URL from your Azure DevOps project (should still be on your clipboard), then click </SPAN><STRONG>Create Git folder</STRONG><SPAN>.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_7-1776187095926.png" style="width: 730px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26008i1CDAB5B3539C6D0B/image-dimensions/730x287?v=v2" width="730" height="287" role="button" title="jim_thorstad_7-1776187095926.png" alt="jim_thorstad_7-1776187095926.png" /></span></P> <P><SPAN>You will notice that Git folders have special properties in the Workspace UI, including these two buttons.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_8-1776187129932.png" style="width: 606px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26009iB65F59BDA86B2ECC/image-dimensions/606x85?v=v2" width="606" height="85" role="button" title="jim_thorstad_8-1776187129932.png" alt="jim_thorstad_8-1776187129932.png" /></span></P> <P><SPAN>Click </SPAN><STRONG>Open in editor</STRONG><SPAN> and notice that a new browser tab opens with your Git folder in a new “focused” </SPAN><STRONG>authoring mode</STRONG><SPAN>. We’ll discuss this in a moment.&nbsp;</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_9-1776187180928.png" style="width: 452px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26010i13DA4733FF040AA0/image-dimensions/452x156?v=v2" width="452" height="156" role="button" title="jim_thorstad_9-1776187180928.png" alt="jim_thorstad_9-1776187180928.png" /></span></P> <P><SPAN>We won’t use this view right now, so close the </SPAN><STRONG>browser tab</STRONG><SPAN> that was just opened.</SPAN></P> <P><SPAN>Now, click the </SPAN><STRONG>Git branch button</STRONG><SPAN> as shown below.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_10-1776187214034.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26011i82EAA6D52F1A4330/image-size/medium?v=v2&amp;px=400" role="button" title="jim_thorstad_10-1776187214034.png" alt="jim_thorstad_10-1776187214034.png" /></span></P> <P><SPAN>In the Git integration UI, click </SPAN><STRONG>Create Branch</STRONG><SPAN>, assign a name to your feature branch (e.g., </SPAN><FONT face="andale mono,times" color="#3366FF">features/initial_work_on_hello_world</FONT><SPAN>), and then click </SPAN><STRONG>Create</STRONG><SPAN>.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_11-1776187253227.png" style="width: 789px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26012i6A40384D2119E024/image-dimensions/789x269?v=v2" width="789" height="269" role="button" title="jim_thorstad_11-1776187253227.png" alt="jim_thorstad_11-1776187253227.png" /></span></P> <P><SPAN>Click the “X” button in the top right corner of the Git integration dialog to close it. Using your Microsoft ENTRA credentials, the Databricks Git integration created the branch in your Azure DevOps Repo and switched you to it in the Workspace UI, as shown below.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_12-1776187294011.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26013i6D51097E8CBE9AA9/image-size/large?v=v2&amp;px=999" role="button" title="jim_thorstad_12-1776187294011.png" alt="jim_thorstad_12-1776187294011.png" /></span></P> <H2><SPAN>Navigating the Workspace UI</SPAN></H2> <P><SPAN>As of April, 2026, there are two UI previews that affect the behavior of the menus shown in this article. Have your workspace Admin go to </SPAN><STRONG>Previews</STRONG><SPAN> and search for “tree”. By default, the new “authoring” context preview (the first one, below) is enabled, and the new tree view for the file browser is not. We recommend enabling both so you can follow along with the article and because they both make working with Git folders and asset bundles easier. Here are the documentation links (as of April, 2026) for reference:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><A href="https://docs.databricks.com/aws/en/workspace/workspace-browser#authoring-contexts" target="_blank" rel="noopener"><SPAN>Authoring context</SPAN></A><SPAN> preview</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><A href="https://docs.databricks.com/aws/en/workspace/workspace-browser#tree-view-of-the-workspace-browser-side-panel-public-preview" target="_blank" rel="noopener"><SPAN>File browser tree view</SPAN></A><SPAN> preview</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>General </SPAN><A href="https://docs.databricks.com/aws/en/workspace/" target="_blank" rel="noopener"><SPAN>Workspace UI documentation</SPAN></A><SPAN> (in case those links above change)</SPAN></LI> </UL> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_13-1776187328636.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26014iB5982225DF6EBD1D/image-size/large?v=v2&amp;px=999" role="button" title="jim_thorstad_13-1776187328636.png" alt="jim_thorstad_13-1776187328636.png" /></span></P> <H2><SPAN>Create and Deploy Your First Bundle to “Dev”</SPAN></H2> <P><SPAN>From your Git folder, click </SPAN><STRONG>Create</STRONG><SPAN> and then choose </SPAN><STRONG>Bundle</STRONG><SPAN>.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_14-1776187353760.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26015iA2B6266BFBA43F80/image-size/large?v=v2&amp;px=999" role="button" title="jim_thorstad_14-1776187353760.png" alt="jim_thorstad_14-1776187353760.png" /></span></P> <P><SPAN>There are different approaches to organizing your development copy of the bundles in your Git folder. A flat list of bundles, as shown below, works fine. You can also create folders under the Git folder and organize your bundles within them.&nbsp;</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_15-1776187376364.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26016iBEEA771F6D6E009B/image-size/medium?v=v2&amp;px=400" role="button" title="jim_thorstad_15-1776187376364.png" alt="jim_thorstad_15-1776187376364.png" /></span></P> <P><SPAN>For the purposes of this blog, enter a name like </SPAN><I><SPAN>hello_world</SPAN></I><SPAN> for the </SPAN><STRONG>Bundle name</STRONG><SPAN> and choose the </SPAN><STRONG>Empty project</STRONG><SPAN> option. After clicking </SPAN><STRONG>Create</STRONG><SPAN>, you will see a list of add-on resources for your bundle. Choose </SPAN><STRONG>Add a new job definition</STRONG><SPAN>, enter a job title like </SPAN><I><SPAN>Update Event Table.&nbsp;</SPAN></I></P> <P><FONT color="#FF0000"><SPAN><U><STRONG>Important:</STRONG> Don’t click</U>&nbsp;<FONT color="#000000">the</FONT>&nbsp;<FONT color="#000000">Add and deploy button</FONT></SPAN><FONT color="#000000"><SPAN>; instead, <STRONG>click the down arrow</STRONG> and choose </SPAN><STRONG>Only add, deploy later</STRONG><SPAN>. </SPAN></FONT></FONT><SPAN>If you accidentally clicked Add and deploy, it's ok, but you have deployed a job to dev that doesn’t have anything to do yet.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_16-1776187458541.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26017iA32BA994C50CAFEF/image-size/large?v=v2&amp;px=999" role="button" title="jim_thorstad_16-1776187458541.png" alt="jim_thorstad_16-1776187458541.png" /></span></P> <H2><SPAN>Add Some Code and Configure Your Job</SPAN></H2> <P><SPAN>Let’s add some code to our project to see the CI/CD process in action. For the purposes of this blog, you can download the</SPAN> <STRONG>Hello_World.ipynb</STRONG> <SPAN>notebook from this location and then import it into your bundle folder as shown below.</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><A href="https://github.com/databricks-solutions/databricks-dab-examples/blob/main/knowledge-base/azure-devops-cicd-example/Hello_World.ipynb" target="_blank" rel="noopener"><SPAN>https://github.com/databricks-solutions/databricks-dab-examples/blob/main/knowledge-base/azure-devops-cicd-example/Hello_World.ipynb</SPAN></A><SPAN>&nbsp;&nbsp;</SPAN></LI> </UL> <P><SPAN>Be sure to import it into the </SPAN><STRONG>hello_world</STRONG><SPAN> asset bundle folder (</SPAN><SPAN>not</SPAN><SPAN> the resources folder), as shown below.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_17-1776187488097.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26018i9859F21B3D51CC35/image-size/large?v=v2&amp;px=999" role="button" title="jim_thorstad_17-1776187488097.png" alt="jim_thorstad_17-1776187488097.png" /></span></P> <P><SPAN>While still on the </SPAN><STRONG>job.yml</STRONG><SPAN> file, uncomment lines 8-19 in the job YAML file using the </SPAN><STRONG>Toggle line comment </STRONG><SPAN>key combination (you can get a list of keyboard shortcuts from the context menu, as shown below).&nbsp; After uncommenting lines, make the two changes shown in yellow. Be aware that paths are case sensitive. You have renamed the first task in your job and pointed it to the notebook you just imported.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_18-1776187507876.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26019iB620B8079CB9EEDB/image-size/large?v=v2&amp;px=999" role="button" title="jim_thorstad_18-1776187507876.png" alt="jim_thorstad_18-1776187507876.png" /></span></P> <P><SPAN>Open the <STRONG>Hello_World</STRONG> notebook and note a couple of things. The first cell defines a widget called “catalog” with no default. The notebook you imported has rendered the catalog widget at the top, but it is empty, so you can fill in the catalog you want to write to. For this lab, we’ll create a catalog called </SPAN><FONT face="andale mono,times" color="#3366FF">bu1_dev</FONT><SPAN>.</SPAN></P> <P><SPAN>Enter your value for the “</SPAN><STRONG>catalog”</STRONG><SPAN> widget field as shown below. Notice that there is also a widget called “</SPAN><STRONG>job_run_id</STRONG><SPAN>” that has no default value.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_19-1776187530474.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26020i27EC6F5734977F37/image-size/large?v=v2&amp;px=999" role="button" title="jim_thorstad_19-1776187530474.png" alt="jim_thorstad_19-1776187530474.png" /></span></P> <P><SPAN>Go ahead and run the notebook now by clicking the </SPAN><STRONG>Run all </STRONG><SPAN>button at the top. You should use </SPAN><STRONG>Serverless</STRONG><SPAN> compute for this article so you don't have to wait for a cluster to start.&nbsp;</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_20-1776187551646.png" style="width: 542px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26021i0481264A64A5B549/image-dimensions/542x141?v=v2" width="542" height="141" role="button" title="jim_thorstad_20-1776187551646.png" alt="jim_thorstad_20-1776187551646.png" /></span></P> <P><SPAN>Notice in the output and the Catalog Explorer that one row was inserted, including the time and the user who ran the code.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_21-1776187579945.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26022iE5850F7B4BE81881/image-size/large?v=v2&amp;px=999" role="button" title="jim_thorstad_21-1776187579945.png" alt="jim_thorstad_21-1776187579945.png" /></span></P> <P><SPAN>In the workspace, click the bundle </SPAN><STRONG>deploy </STRONG><SPAN>button (the “rocket ship”), wait until the </SPAN><STRONG>Bundle resources </STRONG><SPAN>list has finished loading, and then click the </SPAN><STRONG>Deploy</STRONG><SPAN> button.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_22-1776187600169.png" style="width: 635px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26023i70359989C08C0B69/image-dimensions/635x269?v=v2" width="635" height="269" role="button" title="jim_thorstad_22-1776187600169.png" alt="jim_thorstad_22-1776187600169.png" /></span></P> <P><SPAN>Click </SPAN><STRONG>Deploy</STRONG><SPAN> again to confirm your updated Job definition should be deployed.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_23-1776187631019.png" style="width: 501px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26024iB7BB67DC0AE32B20/image-dimensions/501x206?v=v2" width="501" height="206" role="button" title="jim_thorstad_23-1776187631019.png" alt="jim_thorstad_23-1776187631019.png" /></span></P> <P><SPAN>Congratulations!&nbsp; You should see a message that you successfully deployed to your dev target.&nbsp;</SPAN></P> <P><SPAN>Click <STRONG>Workspace</STRONG> in the side navigation to see what got deployed and where.&nbsp;</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_24-1776187657685.png" style="width: 191px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26025iB0A85D9C9170392F/image-dimensions/191x146?v=v2" width="191" height="146" role="button" title="jim_thorstad_24-1776187657685.png" alt="jim_thorstad_24-1776187657685.png" /></span></P> <P><SPAN>Databricks deploys the asset bundle to your </SPAN><FONT face="andale mono,times" color="#3366FF">Home/.bundle</FONT> <SPAN>folder. This is represented by the </SPAN><STRONG>dev deployment environment</STRONG><SPAN> highlighted in blue below. At this point, nothing is in Azure DevOps version control.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_25-1776187687791.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26026iF0F72E8EEC2026DA/image-size/large?v=v2&amp;px=999" role="button" title="jim_thorstad_25-1776187687791.png" alt="jim_thorstad_25-1776187687791.png" /></span></P> <P><SPAN>Click </SPAN><STRONG>Jobs &amp; Pipelines</STRONG><SPAN> in the side navigation.</SPAN></P> <P><SPAN>Your bundle deployment also created a job. Notice that the job title is prepended with</SPAN> <FONT color="#3366FF"><FONT face="andale mono,times">[dev</FONT> <I>your_user_id</I>]</FONT><SPAN>.&nbsp;</SPAN></P> <P><SPAN>Open the job and click the </SPAN><STRONG>Tasks</STRONG><SPAN> tab. Review the information below.&nbsp;</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_26-1776187714667.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26027iE92C20F3D313385D/image-size/large?v=v2&amp;px=999" role="button" title="jim_thorstad_26-1776187714667.png" alt="jim_thorstad_26-1776187714667.png" /></span></P> <P><SPAN>Messages (1) and (4) state that this job was created by a bundle deployment and that you shouldn’t edit this job in the UI (you can edit it, but your edits will be wiped out the next time you deploy your bundle). The name of your notebook task (2) matches what you edited back in the <FONT face="andale mono,times" color="#3366FF">Update Event Table.job.yml</FONT> file. The notebook path (3) you specified in the job.yml file is being used. The job is set to run as you (5). Currently, no job parameters (6) are shown – we will work on this next. There is a </SPAN><STRONG>Disconnect from source</STRONG><SPAN> (7) button in the UI.</SPAN></P> <P><STRONG>Source-linked deployment </STRONG><SPAN>is an optional feature of bundle deployment and is enabled by default for the dev deployment. This is a convenient feature because your dev job references the notebook from your </SPAN><FONT face="andale mono,times" color="#3366FF">Home/bu1-team1/hello_world</FONT><SPAN> folder in the workspace. A copy isn’t pushed into the </SPAN><STRONG>bundle deployment folder</STRONG><SPAN> for dev (<FONT face="andale mono,times" color="#3366FF">Home/.bundle/hello_world</FONT>). What’s convenient about source-linked deployment in dev is that you can make iterative fixes to your notebook and re-run your Job from the Jobs UI </SPAN><STRONG>without having to redeploy your asset bundle</STRONG><SPAN>.</SPAN></P> <P><SPAN>Go back to the </SPAN><STRONG>Runs</STRONG><SPAN> tab and click the </SPAN><STRONG>Run now</STRONG><SPAN> button. Use </SPAN><STRONG>Serverless job compute</STRONG><SPAN> so you don’t have to wait for a job cluster to start.&nbsp;</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_27-1776187740391.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26028iB1A82DFB1384E819/image-size/large?v=v2&amp;px=999" role="button" title="jim_thorstad_27-1776187740391.png" alt="jim_thorstad_27-1776187740391.png" /></span></P> <P><SPAN>The job will fail because the notebook can’t find a value for the catalog variable. When you ran the notebook manually, you typed a value into the catalog widget, but when the notebook ran from a job, the catalog widget was empty, and there was no default value specified in the first cell either.</SPAN></P> <P><SPAN>We don’t want a default value for the catalog because we intend to run this notebook across dev, stage, and prod environments. We want to send specific catalog names to the job based on the environment it's running in. Let’s configure that next.</SPAN></P> <H2><SPAN>Parameterizing the Catalog Name Passed to Your Notebook</SPAN></H2> <P><SPAN>In this section, you’ll be working in two files, which you’ll download in a moment. But let’s first understand how they work together:</SPAN></P> <OL> <LI style="font-weight: 400;" aria-level="1"><STRONG>Bundle configuration (databricks.yml) </STRONG><SPAN>- You’ll define a </SPAN><A href="https://docs.databricks.com/aws/en/dev-tools/bundles/reference#variables" target="_blank" rel="noopener"><STRONG>custom variable</STRONG></A> <I><SPAN>catalog_var</SPAN></I> <SPAN>and some environment-specific values for it (</SPAN><STRONG>bu1_dev</STRONG><SPAN>, </SPAN><STRONG>bu1_stg</STRONG><SPAN>, and </SPAN><STRONG>bu1_prd</STRONG><SPAN>). This will also involve defining our deployment environment </SPAN><STRONG>targets</STRONG><SPAN> (dev, stage, and prod).</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>Job configuration (job.yml)</STRONG><SPAN> - You’ll define a </SPAN><A href="https://docs.databricks.com/aws/en/jobs/job-parameters" target="_blank" rel="noopener"><STRONG>job parameter</STRONG></A><I><SPAN> catalog </SPAN></I><SPAN>to accept the value from the bundle variable during deployment, and we’ll configure a job task </SPAN><A href="https://docs.databricks.com/aws/en/dev-tools/bundles/job-task-types" target="_blank" rel="noopener"><STRONG>base_parameter</STRONG></A><SPAN> so the job parameter is available to our notebook.&nbsp;</SPAN></LI> </OL> <P><SPAN>For more information on these topics, click the links behind the terms above. Visually, the variable/parameter wiring looks like this:</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_28-1776187794608.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26029i27577AF80F84B837/image-size/large?v=v2&amp;px=999" role="button" title="jim_thorstad_28-1776187794608.png" alt="jim_thorstad_28-1776187794608.png" /></span></P> <P><SPAN>As you can see, the bundle configuration file (<FONT face="andale mono,times" color="#3366FF">databricks.yml</FONT>) contains general setup information on lines 1-12 and environment </SPAN><STRONG>targets</STRONG><SPAN> starting on line 14. Each target (dev, staging, and prod) defines the bundle's characteristics for that environment. This includes how to access the Databricks workspace, which variable names/values to pass, and which </SPAN><STRONG>deployment</STRONG> <STRONG>mode</STRONG><SPAN> to use for the environment.</SPAN></P> <P><SPAN>We define a bundle variable </SPAN><STRONG>catalog_var</STRONG><SPAN> and set it to <FONT face="andale mono,times" color="#3366FF">bu1_dev</FONT> when deploying to the “dev” target environment. We’ll set it to <FONT face="andale mono,times" color="#3366FF">bu1_stg</FONT> for the staging environment, and so on. These environmental targets are defined by you. You can have as many as you like and name them however you like. However, </SPAN><STRONG>the bundle target </STRONG><STRONG><I>name</I></STRONG><STRONG> must match the value specified in your Azure DevOps pipeline </STRONG><SPAN>for the bundle validate and deploy steps, as you will see.</SPAN></P> <P><SPAN>The deployment mode setting has two values: development and production. The dev target uses the </SPAN><I><SPAN>development</SPAN></I><SPAN> mode, whereas staging and prod both use the </SPAN><I><SPAN>production</SPAN></I><SPAN> mode. Development mode prefixes your deployed jobs with </SPAN><FONT face="andale mono,times" color="#3366FF">[dev &lt;your_user_name&gt;]</FONT> <SPAN>so that they are easier to distinguish in the event you and your colleagues are both working on separate copies of the same bundle/pipeline in dev. Development mode also enables </SPAN><STRONG>source-linked deployment</STRONG><SPAN> as previously described, making it easier to iterate on your bundle wiring and notebook code. Here’s a complete reference for the behavior of the bundle </SPAN><FONT face="andale mono,times" color="#3366FF"><STRONG>mode: </STRONG></FONT><SPAN>setting.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_29-1776187864368.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26030i38BF44891AE302EE/image-size/large?v=v2&amp;px=999" role="button" title="jim_thorstad_29-1776187864368.png" alt="jim_thorstad_29-1776187864368.png" /></span></P> <P><SPAN>To save you time, here is the completed code for the example we are following. Copy these into your Event Table Job.yml and databricks.yml files.</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN><A href="https://github.com/databricks-solutions/databricks-dab-examples/tree/main/knowledge-base/azure-devops-example/resources/Update_Event_Table.job.yml" target="_blank" rel="noopener">https://github.com/databricks-solutions/databricks-dab-examples/tree/main/knowledge-base/azure-devops-example/resources/Update_Event_Table.job.yml</A>&nbsp;</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN><A href="https://github.com/databricks-solutions/databricks-dab-examples/tree/main/knowledge-base/azure-devops-example/databricks.yml" target="_blank" rel="noopener">https://github.com/databricks-solutions/databricks-dab-examples/tree/main/knowledge-base/azure-devops-example/databricks.yml</A>&nbsp;</SPAN></LI> </UL> <H3><STRONG>Home/bu1-team1/hello_world/resources/Update Event Table.job.yml</STRONG></H3> <P><SPAN>Specify your email in the highlighted line below.&nbsp;</SPAN></P> <P><SPAN>Note that the code includes a second parameter to capture the </SPAN><STRONG>job run ID</STRONG><SPAN> so we can write it into the event table. The syntax</SPAN> <FONT face="andale mono,times" color="#3366FF">{{job.run_id}} </FONT><SPAN>captures the run ID dynamically. We’re appending that to the string “job_run_id_”, which you will see when this parameter is written into the table. You can change the parameter definition and remove the prefix if you wish. Because this parameter is defined dynamically by the job, you do not need to create a bundle variable for it as you did with catalog_var.</SPAN></P> <P>&nbsp;</P> <PRE><SPAN>resources:</SPAN><BR /><SPAN>&nbsp;&nbsp;jobs:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;Update_Event_Table:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;name: Update Event Table</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;queue:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;enabled: true</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;parameters:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- name: catalog</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;default: ${var.catalog_var}</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- name: job_run_id</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;default: "job_run_id_{{job.run_id}}"</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;tasks:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- task_key: notebook_task_1</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;notebook_task:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;notebook_path: ../Hello_World.ipynb</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;base_parameters:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;catalog: ${var.catalog_var}</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;job_run_id: "job_run_id_{{job.run_id}}"</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;email_notifications:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;on_failure:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- </SPAN><FONT color="#FF0000"><SPAN>your_email@example.com</SPAN></FONT><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;trigger:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;periodic:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;interval: 1</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;unit: DAYS</SPAN></PRE> <H3><STRONG>Home/bu1-team1/hello_world/databricks.yml</STRONG></H3> <P><SPAN>Edit the workspace URLs to match your environment and change the catalog names if necessary. Note that we’re using the same workspace for </SPAN><STRONG>dev</STRONG><SPAN> and </SPAN><STRONG>staging</STRONG><SPAN> in this blog, but you can use three separate workspaces for dev/staging/prod if you wish. Just make the edits below.</SPAN></P> <P>&nbsp;</P> <PRE><SPAN># Declarative Automation Bundle (DAB) configuration; more info here:&nbsp;</SPAN><BR /><SPAN># https://docs.databricks.com/en/dev-tools/bundles/settings.html</SPAN><BR /><SPAN>bundle:</SPAN><BR /><SPAN>&nbsp;&nbsp;name: "hello_world"</SPAN><BR /><BR /><SPAN>include:</SPAN><BR /><SPAN>&nbsp;&nbsp;- resources/*.yml</SPAN><BR /><SPAN>&nbsp;&nbsp;- resources/*/*.yml</SPAN><BR /><SPAN># Define custom variables; more info here:&nbsp;</SPAN><BR /><SPAN># https://docs.databricks.com/en/dev-tools/bundles/reference#variables</SPAN><BR /><SPAN>variables:</SPAN><BR /><SPAN>&nbsp;&nbsp;catalog_var:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;description: Custom variable containing the target catalog</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;default: ""</SPAN><BR /><BR /><SPAN>targets:</SPAN><BR /><SPAN>&nbsp;&nbsp;# TODO - adjust the catalog names and workspace URLs to match your deployment</SPAN><BR /><SPAN>&nbsp;&nbsp;# The "dev" target used by the bundle deploy feature</SPAN><BR /><SPAN>&nbsp;&nbsp;dev:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;mode: development</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;default: true</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;variables:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;catalog_var: </SPAN><SPAN>bu1_dev</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;workspace:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;host: https://<FONT color="#FF0000">adb-###########.#.</FONT></SPAN><SPAN>azuredatabricks.net</SPAN><BR /><BR /><SPAN>&nbsp;&nbsp;# The "staging" target used by Azure DevOps</SPAN><BR /><SPAN>&nbsp;&nbsp;staging:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;mode: production</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;variables:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;catalog_var: </SPAN><SPAN>bu1_stg</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;workspace:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;host: https://<FONT color="#FF0000">adb-</FONT></SPAN><FONT color="#FF0000"><SPAN>###########.#</SPAN></FONT><SPAN><FONT color="#FF0000">.</FONT>azuredatabricks.net</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;root_path: /Workspace/devops/staging/bu1-team1/.bundle/${bundle.name}</SPAN><BR /><BR /><SPAN>&nbsp;&nbsp;# The "prod" target used by Azure DevOps</SPAN><BR /><SPAN>&nbsp;&nbsp;prod:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;mode: production</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;variables:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;catalog_var: </SPAN><SPAN>bu1_prd</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;workspace:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;host: https://<FONT color="#FF0000">adb-###########.#</FONT></SPAN><SPAN><FONT color="#FF0000">.</FONT>azuredatabricks.net</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;root_path: /Workspace/devops/prod/bu1-team1/.bundle/${bundle.name}</SPAN></PRE> <P><U><STRONG>Important:</STRONG></U> <SPAN>After updating the workspace URLs in your databricks.yml file, please click the </SPAN><STRONG>reload</STRONG><SPAN> button in your web browser.&nbsp;</SPAN></P> <P><SPAN>This is necessary because we sourced the file with placeholder text (##########) for the workspace URL, and the bundle UI sees it as soon as you make the edit. This confuses it in the next step, as it will think there are no valid targets to deploy to. So after making your edits and reloading the workspace UI in your web browser, continue to the next step.</SPAN></P> <P><SPAN>Click the bundle <STRONG>deploy </STRONG>button and then click the <STRONG>Target</STRONG> pulldown menu.&nbsp;</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_0-1776191070143.png" style="width: 476px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26032i8738D90DFE383DA2/image-dimensions/476x294?v=v2" width="476" height="294" role="button" title="jim_thorstad_0-1776191070143.png" alt="jim_thorstad_0-1776191070143.png" /></span></P> <P><SPAN>Notice the message saying that deployment to other workspaces is not supported. Actually, this means the bundle deploy feature is designed only to deploy locally, to the workspace you are in.&nbsp;</SPAN></P> <P><SPAN>We will only use the bundle deploy button to push the bundle locally to our </SPAN><I><SPAN>dev</SPAN></I><SPAN> target. We will use Azure DevOps to deploy our bundles to the </SPAN><I><SPAN>stage</SPAN></I><SPAN> and </SPAN><I><SPAN>prod</SPAN></I><SPAN> environments.</SPAN></P> <P><SPAN>Press the </SPAN><STRONG>Escape (ESC) key </STRONG><SPAN>to close the drop-down menu and then click the </SPAN><STRONG>Deploy</STRONG><SPAN> button.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_1-1776191123230.png" style="width: 195px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26033i48EB32C184370EB2/image-dimensions/195x125?v=v2" width="195" height="125" role="button" title="jim_thorstad_1-1776191123230.png" alt="jim_thorstad_1-1776191123230.png" /></span></P> <P><SPAN>Click </SPAN><STRONG>Deploy</STRONG><SPAN> again and review the output messages below.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_2-1776191146067.png" style="width: 766px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26034i53152BBBFB7132D8/image-dimensions/766x290?v=v2" width="766" height="290" role="button" title="jim_thorstad_2-1776191146067.png" alt="jim_thorstad_2-1776191146067.png" /></span></P> <P><SPAN>After your DAB deploys to dev, go to your Jobs page and click the run button to start your job. After you start it, you can click your job and see it running.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_3-1776191176189.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26035iEB09146C9FA33806/image-size/large?v=v2&amp;px=999" role="button" title="jim_thorstad_3-1776191176189.png" alt="jim_thorstad_3-1776191176189.png" /></span></P> <P><SPAN>When it finishes, click your job and notice that the catalog and job_id parameters were created and populated as expected, and your code added another row to the <FONT face="andale mono,times" color="#3366FF">bu1_dev.default.test_events</FONT> table.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_4-1776191229954.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26036i0CB7EB0C7FCFD901/image-size/large?v=v2&amp;px=999" role="button" title="jim_thorstad_4-1776191229954.png" alt="jim_thorstad_4-1776191229954.png" /></span></P> <P><SPAN>With your code working in dev, let’s check it into Git.&nbsp;</SPAN></P> <P><SPAN>Click the Workspaces (the “folder”) button and take a few minutes to familiarize yourself with the new Workspace UI. Click the “All files” button and then go back to the “Folder” button next to it. Notice that this simply toggles whether you are in the DAB bundle folder. Also, notice the open file tabs on the right. It’s nice to be able to have multiple files open at one time, even in different folders/subfolders, and switch among them without having to navigate the folder structure on the left.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_5-1776191258090.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26037i840065325E0C22BC/image-size/large?v=v2&amp;px=999" role="button" title="jim_thorstad_5-1776191258090.png" alt="jim_thorstad_5-1776191258090.png" /></span></P> <P><SPAN>When you are ready, click the Git button next to your bundle folder.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_6-1776191292941.png" style="width: 332px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26038i85443CC80AB9F311/image-dimensions/332x151?v=v2" width="332" height="151" role="button" title="jim_thorstad_6-1776191292941.png" alt="jim_thorstad_6-1776191292941.png" /></span></P> <P><SPAN>Go ahead and add some comments and click </SPAN><STRONG>Commit &amp; Push</STRONG><SPAN> for your first DAB project!</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_7-1776191317552.png" style="width: 693px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26039iF8DE4DD5AEABEC4E/image-dimensions/693x566?v=v2" width="693" height="566" role="button" title="jim_thorstad_7-1776191317552.png" alt="jim_thorstad_7-1776191317552.png" /></span></P> <P><SPAN>In the message at the top of the screen, click the create a </SPAN><STRONG>pull request </STRONG><SPAN>link to create your pull request here, for convenience. This takes you right into Azure DevOps, where you can fill in the details for your Pull request!</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_8-1776191354102.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26040iF81045028EF2CD03/image-size/large?v=v2&amp;px=999" role="button" title="jim_thorstad_8-1776191354102.png" alt="jim_thorstad_8-1776191354102.png" /></span></P> <P><SPAN>Click </SPAN><STRONG>Create</STRONG><SPAN> to create your PR and then stop here.</SPAN></P> <P><SPAN>Let’s go set up the Azure DevOps deployment pipeline. This way, when we go into DevOps to approve and merge your PR, we should see our new DevOps pipeline automatically deploy your initial hello world code to staging.</SPAN></P> <H1><SPAN>Part 3 - Creating the Azure DevOps Deployment Pipeline for CI/CD</SPAN></H1> <P><SPAN>As a review, our CI/CD requirements are:</SPAN></P> <OL> <LI><SPAN>Approved developer PRs should be merged into main and automatically deployed to staging.&nbsp;</SPAN><SPAN>Here, the merged code can be tested in the broader context of the data platform, potentially alongside other approved changes awaiting promotion to prod.&nbsp;</SPAN></LI> <LI><SPAN>Then, after further approval, the main branch is deployed to prod.&nbsp;</SPAN></LI> </OL> <P><SPAN>We’ll begin by setting up the staging process, which involves creating a DevOps </SPAN><STRONG>service connection</STRONG><SPAN>, mapping it to an Entra ID-managed Databricks </SPAN><STRONG>service principal </STRONG><SPAN>(SP), creating a workspace </SPAN><STRONG>deployment folder</STRONG><SPAN> for staging, and granting the SP permissions on the folder and the </SPAN><FONT face="andale mono,times" color="#3366FF">bu1_stg</FONT><STRONG> catalog</STRONG><SPAN> in Unity Catalog. We’ll repeat this process for prod. Then we’ll create the DevOps </SPAN><STRONG>variable groups</STRONG><SPAN>, prod </SPAN><STRONG>environment</STRONG><SPAN> gate/approvers, and </SPAN><STRONG>pipeline</STRONG><SPAN> to pull it all together. After that, we can test it out!</SPAN></P> <P><SPAN>Here’s an overview of the steps that you’ll perform twice (once for staging and once for prod), except where noted:</SPAN></P> <TABLE> <TBODY> <TR> <TD> <P><SPAN>Step</SPAN></P> </TD> <TD> <P><SPAN>Where</SPAN></P> </TD> <TD> <P><SPAN>What</SPAN></P> </TD> <TD> <P><SPAN>Why</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>1</SPAN></P> </TD> <TD> <P><SPAN>Azure DevOps &gt; Project Settings</SPAN></P> </TD> <TD> <P><SPAN>Create the bu1-team1 Service Connection</SPAN></P> </TD> <TD rowspan="2"> <P><SPAN>Facilitates a secure and flexible connection between DevOps and Azure Databricks for deploying bundles to the staging and prod.&nbsp;</SPAN></P> <BR /> <P><SPAN>For a comparison to the newer Databricks OIDC-based approach, see the topic at the end of this article.</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>2</SPAN></P> </TD> <TD> <P><SPAN>Azure Portal</SPAN></P> </TD> <TD> <P><SPAN>Create the bu1-team1-cicd App Registration and Federated credential</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>3</SPAN></P> </TD> <TD> <P><SPAN>Databricks workspace &gt; Settings</SPAN></P> </TD> <TD> <P><SPAN>Create the bu1-team1-cicd Service Principal&nbsp;</SPAN></P> </TD> <TD> <P><SPAN>This will be the owner of the resources deployed by DevOps; you need to give this principal write access to the deployment folder and privileges in Unity Catalog</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>4</SPAN></P> </TD> <TD> <P><SPAN>Databricks workspace &gt; Folders</SPAN></P> </TD> <TD> <P><SPAN>Create your SP and any admin group permissions on the /staging/bu1-team1 folder</SPAN></P> </TD> <TD> <P><SPAN>By deploying into team-based folders, you have the ability to set permissions so each team can view their own deployed code if necessary</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>5</SPAN></P> </TD> <TD> <P><SPAN>Databricks workspace &gt; Catalog</SPAN></P> </TD> <TD> <P><SPAN>Give the SP and other groups appropriate permissions in Unity Catalog.&nbsp;</SPAN></P> </TD> <TD> <P><SPAN>Tables and schemas created by the SP are owned by the SP.</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>6</SPAN></P> </TD> <TD> <P><SPAN>Azure DevOps &gt; Library&nbsp;</SPAN></P> </TD> <TD> <P><SPAN>Create a Variable group for each stage</SPAN></P> </TD> <TD> <P><SPAN>These hold the variable DATABRICKS_HOST with the URL to the workspace for the stage</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>7</SPAN></P> </TD> <TD> <P><SPAN>Azure DevOps &gt; Environments&nbsp;</SPAN></P> </TD> <TD> <P><SPAN>You will create this only for “prod”</SPAN></P> </TD> <TD> <P><SPAN>This will be used to set up the approval gate for prod deployment</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>8</SPAN></P> </TD> <TD> <P><SPAN>Azure DevOps &gt; Repo</SPAN></P> </TD> <TD> <P><SPAN>Create a root folder called “/.cicd” or similar for the pipeline code.</SPAN></P> </TD> <TD> <P><SPAN>This provides separation between your pipeline and bundle code. Consider requiring special approvers for code changes to files in this root folder.</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>9</SPAN></P> </TD> <TD> <P><SPAN>Azure DevOps &gt; Pipelines</SPAN></P> </TD> <TD> <P><SPAN>Create a release pipeline for your hello_world project</SPAN></P> </TD> <TD> <P><SPAN>This will deploy your code to staging and prod after the required approvals.&nbsp;</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>10</SPAN></P> </TD> <TD> <P><SPAN>Test it out!</SPAN></P> </TD> <TD> <P><SPAN>Check the messages written into your event table in the staging and prod catalogs</SPAN></P> </TD> <TD> <P><SPAN>You can consider how to organize your bundles and deployment pipelines. A pipeline can deploy multiple bundles, or you can keep them 1:1 for finer control.</SPAN></P> </TD> </TR> </TBODY> </TABLE> <P>&nbsp;</P> <H2><SPAN>Create the Staging Service Connection and App Registration</SPAN></H2> <P><SPAN>Go to your DevOps project’s</SPAN><STRONG> Project settings &gt; Pipelines &gt; Service connections</STRONG><SPAN> and click </SPAN><STRONG>Create service connection</STRONG><SPAN>.&nbsp;</SPAN></P> <P><SPAN>Choose the </SPAN><STRONG>Azure Resource Manager</STRONG><SPAN> option and click </SPAN><STRONG>Next</STRONG><SPAN>.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_9-1776191487263.png" style="width: 728px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26041iEFAEAC300A1C495E/image-dimensions/728x418?v=v2" width="728" height="418" role="button" title="jim_thorstad_9-1776191487263.png" alt="jim_thorstad_9-1776191487263.png" /></span></P> <P><FONT color="#FF0000"><STRONG>Important! </STRONG></FONT><SPAN><FONT color="#FF0000">&nbsp;</FONT>For </SPAN><STRONG>Identity type,</STRONG><SPAN> you’ll see two options.&nbsp;</SPAN></P> <P><SPAN>Choose the </SPAN><STRONG>App registration or managed identity (manual) </STRONG><SPAN>option highlighted below (</SPAN><I><SPAN>do not choose</SPAN></I><SPAN> the one tagged as recommended).&nbsp;</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_10-1776191527970.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26042i64FE2F4A9422ABAA/image-size/medium?v=v2&amp;px=400" role="button" title="jim_thorstad_10-1776191527970.png" alt="jim_thorstad_10-1776191527970.png" /></span></P> <P><SPAN>This lets you clearly name the Azure </SPAN><STRONG>App Registration </STRONG><SPAN>backing your DevOps </SPAN><STRONG>Service Connection</STRONG><SPAN>. This is important because the name of your Azure App Registration is synchronized with the name of your Databricks </SPAN><STRONG>Service Principal, </STRONG><SPAN>and this is what your data team will be searching for when granting permissions on your Unity Catalog and Workspace resources.&nbsp;</SPAN></P> <P><SPAN>If you choose the other option, the App Registration name will be automatically generated by Azure as</SPAN> <FONT face="andale mono,times" color="#3366FF">&lt;devops_org&gt;-&lt;project&gt;-&lt;GUID&gt;</FONT><SPAN>, with no indication of whether it is for staging or prod. You cannot rename it, and this will make it hard to assign permissions later. The manual approach we recommend in this article involves a few more steps, but it is your best option as of April 2026.</SPAN></P> <P><SPAN>Note: Both approaches involve creating an App registration in your Azure portal, so if you don’t have the permissions to do this, you’ll get an error and will need to work on these steps with someone who does.&nbsp;</SPAN></P> <P><SPAN>After choosing </SPAN><STRONG>App registration or managed identity (manual),</STRONG><SPAN> fill in the </SPAN><STRONG>Service Connection Name (e.g., </STRONG><FONT face="andale mono,times" color="#3366FF"><SPAN>bu1-team1-staging-svc-conn</SPAN></FONT><STRONG>)</STRONG><SPAN>, description, and your Directory (tenant) ID under the </SPAN><STRONG>Part 1: Basics</STRONG><SPAN> section, and then click </SPAN><STRONG>Next</STRONG><SPAN>.&nbsp;</SPAN></P> <P><SPAN>This takes you to </SPAN><STRONG>Part 2: App registration details</STRONG><SPAN>.</SPAN></P> <P><SPAN>Tip:</SPAN><SPAN> to find your tenant ID, click your user ID circle in the DevOps portal banner bar and then click the </SPAN><STRONG>Switch directory</STRONG><SPAN> link. The tenant ID will be the string under the current directory.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_11-1776191563270.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26043iAE1BB5F286D77756/image-size/medium?v=v2&amp;px=400" role="button" title="jim_thorstad_11-1776191563270.png" alt="jim_thorstad_11-1776191563270.png" /></span></P> <P><SPAN>At this point, you’ll need to keep the Part 2 screen open and open the Azure Portal in another tab. You can follow the steps below and see them illustrated in the image below.</SPAN></P> <OL> <LI style="font-weight: 400;" aria-level="1"><SPAN>Create the </SPAN><STRONG>App Registration</STRONG><SPAN> (e.g., <FONT face="andale mono,times" color="#3366FF">bu1-team1-cicd-staging-app</FONT>) in the Azure portal.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Copy the </SPAN><STRONG>Application ID</STRONG><SPAN> (formerly called client ID) and add it to your DevOps </SPAN><STRONG>service connection Part 2</STRONG><SPAN> field labeled </SPAN><STRONG>Application (client) ID.</STRONG></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>While you are there, fill in your </SPAN><STRONG>Subscription ID</STRONG><SPAN>. You can get the Subscription ID by searching Subscriptions in the Azure Portal.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Enter your </SPAN><STRONG>Subscription name</STRONG><SPAN> - do not click the Verify and save button yet.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Back in the Azure Portal on your App Registration page, click </SPAN><STRONG>Manage</STRONG><SPAN> on the left.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Then click </SPAN><STRONG>Certificates &amp; secrets</STRONG><SPAN>.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Click the </SPAN><STRONG>Federated credentials</STRONG><SPAN> tab.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Click </SPAN><STRONG>Add Credentia</STRONG><SPAN>l.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Choose </SPAN><STRONG>Other issuer</STRONG><SPAN>.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Copy the </SPAN><STRONG>Issuer URL</STRONG><SPAN> from your service connection under </SPAN><STRONG>Part 2</STRONG><SPAN> and paste it in the </SPAN><STRONG>Issuer</STRONG><SPAN> field.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Copy the </SPAN><STRONG>Subject identifier </STRONG><SPAN>and paste it in the </SPAN><STRONG>Value</STRONG><SPAN> field.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Enter a </SPAN><STRONG>name</STRONG><SPAN> for your credential (e.g. <FONT face="andale mono,times" color="#3366FF">bu1-team1-cicd-staging-fed-credential</FONT>)</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Click </SPAN><STRONG>Add</STRONG><SPAN>.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Back on the service connection screen, check the </SPAN><STRONG>Grant access permission to all pipelines</STRONG><SPAN> box. This is okay because it means all DevOps pipelines in our <FONT face="andale mono,times" color="#3366FF">bu1-team</FONT> DevOps project. If you are organizing things differently, you can always grant access to this connection later on.</SPAN></LI> </OL> <P><STRONG>Don’t click Verify and save yet!&nbsp; </STRONG><SPAN>You need to give your App Registration permission to read your Subscription. If you clicked it anyway, look carefully at the error message. You’ll see that your Part 1 work was saved as a draft. You can come back and complete Part 2 after you do the required steps.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_12-1776191605220.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26044i64DA3F4AA9C7F55B/image-size/large?v=v2&amp;px=999" role="button" title="jim_thorstad_12-1776191605220.png" alt="jim_thorstad_12-1776191605220.png" /></span></P> <P><SPAN>Go to the </SPAN><STRONG>Azure Portal</STRONG><SPAN> and find your Subscription. Go to </SPAN><STRONG>Access control (IAM)</STRONG><SPAN> and click </SPAN><STRONG>Add role assignment</STRONG><SPAN>. Click </SPAN><STRONG>Reader</STRONG><SPAN> under Name and then click </SPAN><STRONG>Next</STRONG><SPAN>. Leave the </SPAN><STRONG>Assign access</STRONG><SPAN> to option set to </SPAN><STRONG>User, group, or service principal </STRONG><SPAN>and click the </SPAN><STRONG>Select members</STRONG><SPAN> link. Find your </SPAN><STRONG>service principal </STRONG><SPAN>(note: your App Registration previously created a service principal in the background with the same name, e.g., <FONT face="andale mono,times" color="#3366FF">bu1-team1-cicd-staging-app</FONT>), pick it, and click </SPAN><STRONG>Select</STRONG><SPAN>.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_13-1776191638184.png" style="width: 786px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26045i5F3A553B419647B8/image-dimensions/786x333?v=v2" width="786" height="333" role="button" title="jim_thorstad_13-1776191638184.png" alt="jim_thorstad_13-1776191638184.png" /></span></P> <P><SPAN>Click </SPAN><STRONG>Review + Assign</STRONG><SPAN>.</SPAN></P> <P><SPAN>Now return to Azure DevOps and your </SPAN><STRONG>Edit service connection Part 2</STRONG><SPAN> screen and click the </SPAN><STRONG>Verify and save</STRONG><SPAN> button.</SPAN></P> <P><SPAN>Congratulations!&nbsp; </SPAN><SPAN>That’s a lot of steps, but it's necessary because you need to establish a trust relationship between your Azure DevOps project and your Azure Portal App Registration/Service Principal; these services do not all trust each other by default.&nbsp;</SPAN></P> <P><SPAN>Now we are going to configure trust between our Azure App Registration/Service Principal and the Databricks Service Principal that will run the resources Azure DevOps deployed in your workspace.</SPAN></P> <H2><SPAN>Create the Databricks Service Principal for CI/CD Tasks</SPAN></H2> <P><SPAN>Go to your staging workspace (in this article, we use the <FONT face="andale mono,times" color="#3366FF">bu1-dev</FONT> workspace). Go to Workspace </SPAN><STRONG>Settings</STRONG><SPAN> &gt; </SPAN><STRONG>Identity and access</STRONG><SPAN> &gt; </SPAN><STRONG>Service principals</STRONG><SPAN> and click </SPAN><STRONG>Manage</STRONG><SPAN>. Click </SPAN><STRONG>Add service principal </STRONG><SPAN>and then click the </SPAN><STRONG>Add new</STRONG><SPAN> button.</SPAN></P> <P><SPAN>Under Management, choose </SPAN><STRONG>Microsoft Entra ID managed,</STRONG><SPAN> and then copy the </SPAN><STRONG>Application ID</STRONG><SPAN> from your Azure App Registration over.&nbsp;</SPAN></P> <P><EM>Note: as of April 2026, the UI shown below for the Microsoft Entra ID managed option shows a field for entering the service principal name. While this field allows you to enter and save a value, it will not be retained, as you will see when you go to search for this service principal in the permissions UI throughout Databricks.</EM></P> <P><SPAN>Give it any </SPAN><STRONG>name</STRONG><SPAN> like bu1-team1-cicd-staging-sp (but you’ll see it will be changed back to the name of your App Registration shortly) and click </SPAN><STRONG>Add</STRONG><SPAN>.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_14-1776191753695.png" style="width: 811px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26046i68C7EF413DC21A71/image-dimensions/811x517?v=v2" width="811" height="517" role="button" title="jim_thorstad_14-1776191753695.png" alt="jim_thorstad_14-1776191753695.png" /></span></P> <P><SPAN>Note: It’s possible to use a </SPAN><I><SPAN>single</SPAN></I><SPAN> Azure DevOps Service Connection, Azure App Registration, and Databricks Service Principal for all of your CI/CD processes (including both staging and production deployments). However, you’ll notice we’ve been including the term “staging” in the names of these CI/CD resources we’ve been creating, so you can later add the “prod” versions and have more precise governance over your CI/CD process.</SPAN></P> <P><SPAN>Click your Databricks SP after it's created. It will need the </SPAN><STRONG>Workspace Access</STRONG><SPAN> and </SPAN><STRONG>SQL access</STRONG><SPAN> entitlements, depending on your code. I recommend using Serverless jobs, especially during development, to speed up your trial-and-error CI/CD testing. If you use Serverless jobs, you do not need to assign the </SPAN><STRONG>Allow unrestricted cluster creation</STRONG><SPAN> entitlement.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_15-1776191786351.png" style="width: 356px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26047i22CFA43286233E4E/image-dimensions/356x344?v=v2" width="356" height="344" role="button" title="jim_thorstad_15-1776191786351.png" alt="jim_thorstad_15-1776191786351.png" /></span></P> <P><SPAN>Explore the </SPAN><STRONG>Permissions</STRONG><SPAN> tab. This lets you decide who can manage and use the SP, not what this SP can do in your workspace. You can consider whether a team in the staging environment needs to manually re-run a job for some reason. In this case, give that team the USE permission.</SPAN></P> <P><SPAN>Explore the </SPAN><STRONG>Git integration</STRONG><SPAN> tab. This feature allows your SP to connect with a Git repo, which is needed when the job is configured to pull notebook code from Git at runtime. But we are not using that approach in this blog. We are configuring DevOps to push the notebook code and bundle resources to each environment. Leave Git integration unconfigured for the SP.</SPAN></P> <H2><SPAN>Create the Staging Deployment Folder</SPAN></H2> <P><SPAN>In the workspace, create a staging location where the Azure DevOps pipeline will deploy your code. Then right-click this folder and choose the </SPAN><STRONG>Share (Permissions) </STRONG><SPAN>menu. Search for your SP and grant it </SPAN><STRONG>Can</STRONG> <STRONG>Manage</STRONG><SPAN> permissions, as shown below (remember, as mentioned previously, you are searching for the App Registration you created in the Azure portal, which is synced to the name of the Databricks service principal you just created).&nbsp;</SPAN></P> <P><SPAN>If you would like your development team to see what is deployed here, grant them View permissions. Generally, developers, QA teams, and admins should not have write access to the staging bundle target deployment folder.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_16-1776191820288.png" style="width: 762px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26048iBB0F3794C0A25793/image-dimensions/762x238?v=v2" width="762" height="238" role="button" title="jim_thorstad_16-1776191820288.png" alt="jim_thorstad_16-1776191820288.png" /></span></P> <H2><SPAN>Assigning Unity Catalog Permissions to your Service Principal</SPAN></H2> <P><SPAN>There are a few strategies for creating and managing permissions on catalogs and schemas. In this blog, we create the catalogs manually and assign Data Editor permissions to our SP. We also grant the Manage permission so our SP can drop tables and manage permissions if we were to do that in our job code. Feel free to reconsider and evolve your choices as necessary.</SPAN></P> <P><SPAN>Give your SP permission to create data assets in your </SPAN><STRONG>stage catalog</STRONG><SPAN> as shown below.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_17-1776191855258.png" style="width: 688px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26049iDA5EE4105035AB13/image-dimensions/688x647?v=v2" width="688" height="647" role="button" title="jim_thorstad_17-1776191855258.png" alt="jim_thorstad_17-1776191855258.png" /></span></P> <P><SPAN>Go into the default schema in your staging catalog and notice that many of these permissions are inherited as expected. Look at the permissions on your </SPAN><FONT face="andale mono,times" color="#3366FF">bu1_stg.default</FONT> schema<SPAN>. Notice that your SP can read and write to this schema even though it is not the owner (you are the owner since it was created when you made the catalog).</SPAN></P> <H1><SPAN>Create your Azure DevOps Pipeline (bu1-team1-hello-world-pipeline.yml)</SPAN></H1> <P><SPAN>We want our CI/CD pipeline to have two stages:</SPAN></P> <OL> <LI style="font-weight: 400;" aria-level="1"><STRONG>DeployStaging</STRONG><SPAN> - Runs automatically when code from a feature branch has been approved and merged into the main branch.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>DeployProd</STRONG><SPAN> - Runs after DeployStaging and after an approval that you can configure.</SPAN></LI> </OL> <P><SPAN>We’ll use DevOps variable groups to store the workspace URLs and a DevOps environment to define the approval process that gates our prod deployment.</SPAN></P> <P><SPAN>Go to your </SPAN><STRONG>DevOps project &gt; Pipelines &gt; Library</STRONG><SPAN> and click the </SPAN><STRONG>+ Variable group</STRONG><SPAN> button. Name it </SPAN><STRONG>stagingVariables</STRONG><SPAN> (you’ll reference this in the first stage of your pipeline) and add a variable for </SPAN><STRONG>DATABRICKS_HOST</STRONG><SPAN> along with a URL pointing to the hostname of your staging workspace (bu1-dev in the example for this article). Then click </SPAN><STRONG>Save</STRONG><SPAN>.&nbsp;</SPAN></P> <P><SPAN>Repeat the process for another variable group called </SPAN><STRONG>prodVariables</STRONG><SPAN>, which you will reference in the second stage of your pipeline.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_18-1776191886815.png" style="width: 781px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26050i4725CE906CB12AB6/image-dimensions/781x300?v=v2" width="781" height="300" role="button" title="jim_thorstad_18-1776191886815.png" alt="jim_thorstad_18-1776191886815.png" /></span></P> <P><SPAN>Go to your </SPAN><STRONG>DevOps project &gt; Environments </STRONG><SPAN>and click the </SPAN><STRONG>Create environment</STRONG><SPAN> button. Name it </SPAN><STRONG>prod</STRONG><SPAN> (you’ll reference this in the second stage of your pipeline), leave Resource set to </SPAN><STRONG>None</STRONG><SPAN>, and click </SPAN><STRONG>Create</STRONG><SPAN>.&nbsp;</SPAN></P> <P><SPAN>Next, click the </SPAN><STRONG>Approvals and checks</STRONG><SPAN> tab and click the </SPAN><STRONG>Approvals</STRONG><SPAN> check. For this article, we’ll just pick two approvers, but you can configure this as required by your organization.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_19-1776191913682.png" style="width: 800px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26051i5E359CD85BF19784/image-dimensions/800x279?v=v2" width="800" height="279" role="button" title="jim_thorstad_19-1776191913682.png" alt="jim_thorstad_19-1776191913682.png" /></span></P> <P><SPAN>You do not need to create a staging environment, as we plan to deploy automatically after merging to main with no conditions or approvals.</SPAN></P> <P><SPAN>Go to your </SPAN><STRONG>DevOps project &gt; Repos &gt; Files</STRONG><SPAN> page and create a new folder (e.g., </SPAN><STRONG>.cicd</STRONG><SPAN>) and your Hello World pipeline inside it.&nbsp;</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_20-1776191946563.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26052i69DF7CA4F0DF1C15/image-size/medium?v=v2&amp;px=400" role="button" title="jim_thorstad_20-1776191946563.png" alt="jim_thorstad_20-1776191946563.png" /></span></P> <P><SPAN>Copy the following YAML into it and edit the </SPAN><STRONG>azureSubscription value on lines 33 and 102 </STRONG><SPAN>to match the name of your DevOps Service Connections (yes; its confusing that it needs the service connection name, but it asks for the subscription name). Review the comments and make any other necessary adjustments.</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>You can also download the file from this link - </SPAN><A href="https://github.com/databricks-solutions/databricks-dab-examples/blob/main/knowledge-base/azure-devops-cicd-example/bu1-team1-hello-world-pipeline.yml" target="_blank" rel="noopener"><SPAN>https://github.com/databricks-solutions/databricks-dab-examples/blob/main/knowledge-base/azure-devops-cicd-example/bu1-team1-hello-world-pipeline.yml</SPAN></A><SPAN>&nbsp;</SPAN></LI> </UL> <PRE><SPAN># This triggers the pipeline on a commit to the 'main' branch</SPAN><BR /><SPAN>trigger:</SPAN><BR /><SPAN>&nbsp;&nbsp;branches:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;include:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;- main</SPAN><BR /><BR /><SPAN># Prevents the pipeline from triggering on Pull Requests; only on merges</SPAN><BR /><SPAN>pr: none</SPAN><BR /><BR /><SPAN>stages:</SPAN><BR /><SPAN># -----------------------------------------------</SPAN><BR /><SPAN># Stage 1: Deploy to Staging (automatic on merge)</SPAN><BR /><SPAN># -----------------------------------------------</SPAN><BR /><SPAN># TODO - In the bundle validate/deploy steps below, change "workingDirectory"</SPAN><BR /><SPAN># to the folder containing your databricks.yml and edit the target values if</SPAN><BR /><SPAN># your choice differs from "staging" used in the example below.</SPAN><BR /><SPAN>- stage: DeployStaging</SPAN><BR /><SPAN>&nbsp;&nbsp;displayName: 'Deploy to Staging'</SPAN><BR /><SPAN>&nbsp;&nbsp;variables:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;- group: stagingVariables</SPAN><BR /><SPAN>&nbsp;&nbsp;condition: |</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;eq(variables['Build.SourceBranch'], 'refs/heads/main')</SPAN><BR /><SPAN>&nbsp;&nbsp;jobs:</SPAN><BR /><SPAN>&nbsp;&nbsp;- job: deployStagingJob</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;pool:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;vmImage: ubuntu-latest</SPAN><BR /><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;steps:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;- task: UsePythonVersion@0&nbsp;&nbsp;</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;inputs:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;versionSpec: 3.11&nbsp;&nbsp;</SPAN><BR /><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;- task: AzureCLI@2</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;displayName: 'Get Databricks Token (Staging)'</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;inputs:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# TODO - Change azureSubscription value to the name of your DevOps "staging" Service Connection</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;azureSubscription: </SPAN><FONT color="#FF0000"><SPAN>bu1-team1-staging-svc-conn</SPAN></FONT><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;scriptType: 'bash'</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;scriptLocation: 'inlineScript'</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;failOnStandardError: true</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# Set the token as a pipeline variable for subsequent steps; note that the "echo"</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# line does not print the token to the pipeline logs. The resource ID is the well-known</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# Azure AD application ID for Databricks (same for all tenants).</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;inlineScript: |</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;set -e</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;DATABRICKS_TOKEN=$(az account get-access-token --resource 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d --query "accessToken" -o tsv)&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;echo "##vso[task.setvariable variable=DATABRICKS_TOKEN]$DATABRICKS_TOKEN"<BR /></SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;# Clone the repository so the bundle files are available to the pipeline agent</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;- checkout: self</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;displayName: 'Checkout repository'</SPAN><BR /><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;# Install Python dependencies here if your bundle requires them</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;# For example:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;# - script: pip install -r requirements.txt</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;# &nbsp; displayName: 'Install Python dependencies'&nbsp;</SPAN><BR /><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;- script: |</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;curl -fsSL <A href="https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh" target="_blank" rel="noopener">https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh</A> | sh</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;databricks --version</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;displayName: 'Install Databricks CLI'</SPAN><BR /><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;- script: databricks bundle validate -t staging</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;displayName: 'Validate bundle for the "staging" environment'</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;env:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;DATABRICKS_HOST: $(DATABRICKS_HOST)</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;DATABRICKS_TOKEN: $(DATABRICKS_TOKEN)</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;workingDirectory: $(Build.SourcesDirectory)/hello_world</SPAN><BR /><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;- script: databricks bundle deploy -t staging</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;displayName: 'Deploy bundle to the "staging" environment'</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;env:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;DATABRICKS_HOST: $(DATABRICKS_HOST)</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;DATABRICKS_TOKEN: $(DATABRICKS_TOKEN)</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;workingDirectory: $(Build.SourcesDirectory)/hello_world</SPAN><BR /><BR /><SPAN># -----------------------------------------------</SPAN><BR /><SPAN># Stage 2: Deploy to Production (requires approval in Azure DevOps)</SPAN><BR /><SPAN># -----------------------------------------------</SPAN><BR /><SPAN># TODO - In the bundle validate/deploy steps below, change "workingDirectory"</SPAN><BR /><SPAN># to the folder containing your databricks.yml and edit the target values if</SPAN><BR /><SPAN># your choice differs from "prod" used in the example below.</SPAN><BR /><SPAN>- stage: DeployProd</SPAN><BR /><SPAN>&nbsp;&nbsp;displayName: 'Deploy to Prod'</SPAN><BR /><SPAN>&nbsp;&nbsp;dependsOn: DeployStaging</SPAN><BR /><SPAN>&nbsp;&nbsp;condition: succeeded()</SPAN><BR /><SPAN>&nbsp;&nbsp;variables:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;- group: prodVariables</SPAN><BR /><SPAN>&nbsp;&nbsp;jobs:</SPAN><BR /><SPAN>&nbsp;&nbsp;# A "deployment:" job (rather than a regular "job:") is required to</SPAN><BR /><SPAN>&nbsp;&nbsp;# connect to an Azure DevOps Environment and enable the approval gate.</SPAN><BR /><SPAN>&nbsp;&nbsp;- deployment: deployProdJob</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;displayName: 'Deploy to Prod'</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;pool:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;vmImage: ubuntu-latest</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;# Setup the approval gate. The value for "environment" must match&nbsp;</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;# the Azure DevOps Environment name where approvals are configured.</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;environment: 'prod'</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;strategy:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;runOnce:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;deploy:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;steps:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- checkout: self</SPAN><BR /><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- task: UsePythonVersion@0</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;displayName: 'Use Python 3.11'</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;inputs:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;versionSpec: 3.11</SPAN><BR /><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- task: AzureCLI@2</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;displayName: 'Get Databricks Token (Prod)'</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;inputs:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# TODO - Change azureSubscription value to the name of your DevOps "prod" Service Connection</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;azureSubscription: </SPAN><FONT color="#FF0000"><SPAN>bu1-team1-prod-svc-conn</SPAN></FONT><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;scriptType: 'bash'</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;scriptLocation: 'inlineScript'</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;failOnStandardError: true</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# Set the token as a pipeline variable for subsequent steps; note that the "echo"</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# line does not print the token to the pipeline logs. The resource ID is the well-known</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# Azure AD application ID for Databricks (same for all tenants).</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;inlineScript: |</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;set -e</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;DATABRICKS_TOKEN=$(az account get-access-token --resource 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d --query "accessToken" -o tsv)&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;echo "##vso[task.setvariable variable=DATABRICKS_TOKEN]$DATABRICKS_TOKEN"</SPAN><BR /><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# Install Python dependencies here if your bundle requires them</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# For example:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# - script: pip install -r requirements.txt</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# &nbsp; displayName: 'Install Python dependencies'</SPAN><BR /><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- script: |</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;curl -fsSL <A href="https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh" target="_blank" rel="noopener">https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh</A> | sh</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;databricks --version</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;displayName: 'Install Databricks CLI'</SPAN><BR /><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- script: databricks bundle validate -t prod</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;displayName: 'Validate bundle for the "prod" environment'</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;env:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;DATABRICKS_HOST: $(DATABRICKS_HOST)</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;DATABRICKS_TOKEN: $(DATABRICKS_TOKEN)</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;workingDirectory: $(Build.SourcesDirectory)/hello_world</SPAN><BR /><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- script: databricks bundle deploy -t prod</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;displayName: 'Deploy bundle to the "prod" environment'</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;env:</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;DATABRICKS_HOST: $(DATABRICKS_HOST)</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;DATABRICKS_TOKEN: $(DATABRICKS_TOKEN)</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;workingDirectory: $(Build.SourcesDirectory)/hello_world</SPAN></PRE> <P><SPAN>Click </SPAN><STRONG>Commit</STRONG><SPAN> to save it to the main branch of your repo.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_21-1776192316430.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26053i2AD7E30B04C08767/image-size/large?v=v2&amp;px=999" role="button" title="jim_thorstad_21-1776192316430.png" alt="jim_thorstad_21-1776192316430.png" /></span></P> <P><SPAN>Before making the DevOps pipeline and pointing it to this file, let’s approve and merge the initial Hello World DABs pull request so there is some code to deploy.&nbsp;</SPAN></P> <P><SPAN>In your Azure DevOps project, click </SPAN><STRONG>Repos &gt; Pull requests</STRONG><SPAN> and select the pull request you created earlier from the Databricks Git integration UI.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_22-1776192352175.png" style="width: 687px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26054i2DEB18506BDF2838/image-dimensions/687x230?v=v2" width="687" height="230" role="button" title="jim_thorstad_22-1776192352175.png" alt="jim_thorstad_22-1776192352175.png" /></span></P> <P><SPAN>Click </SPAN><STRONG>Complete</STRONG><SPAN> and then choose your merge options before clicking the </SPAN><STRONG>Complete merge </STRONG><SPAN>button.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_23-1776192375534.png" style="width: 644px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26055i3F8200E21D3E5CA3/image-dimensions/644x306?v=v2" width="644" height="306" role="button" title="jim_thorstad_23-1776192375534.png" alt="jim_thorstad_23-1776192375534.png" /></span></P> <P><SPAN>You can verify that your bundle code is now in the main branch.</SPAN></P> <P><SPAN>Now let’s create the pipeline. Go to your </SPAN><STRONG>project &gt; Pipelines &gt; Pipelines</STRONG><SPAN> menu and click </SPAN><STRONG>Create Pipeline</STRONG><SPAN>. Choose </SPAN><STRONG>Azure Repos Git</STRONG><SPAN>.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_24-1776192394894.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26056iCC7F5DAD5636E244/image-size/large?v=v2&amp;px=999" role="button" title="jim_thorstad_24-1776192394894.png" alt="jim_thorstad_24-1776192394894.png" /></span></P> <P><SPAN>Choose your team repository and then click </SPAN><STRONG>Existing Azure Pipelines YAML file</STRONG><SPAN>.</SPAN></P> <P><SPAN>Enter the path to your pipeline file and click </SPAN><STRONG>Continue</STRONG><SPAN>.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_25-1776192423890.png" style="width: 712px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26057i5CE1C9FAF4CF4867/image-dimensions/712x389?v=v2" width="712" height="389" role="button" title="jim_thorstad_25-1776192423890.png" alt="jim_thorstad_25-1776192423890.png" /></span></P> <P><SPAN>Before you can run the pipeline, you need to either:</SPAN></P> <OL> <LI style="font-weight: 400;" aria-level="1"><SPAN>Repeat the previous steps to create your prod service connection (<FONT face="andale mono,times" color="#3366FF">bu1-team1-prod-svc-conn</FONT>), your Databricks SP and deployment folder on the prod workspace, and the permissions for the prod catalog, or</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Comment out all Stage 2 lines in your pipeline to avoid validation errors.</SPAN></LI> </OL> <H2><SPAN>Deploying to Staging</SPAN></H2> <P><SPAN>Once you have decided, click </SPAN><STRONG>Run</STRONG><SPAN>.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_26-1776192461650.png" style="width: 801px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26058i072D2927AA7B964D/image-dimensions/801x226?v=v2" width="801" height="226" role="button" title="jim_thorstad_26-1776192461650.png" alt="jim_thorstad_26-1776192461650.png" /></span></P> <P><SPAN>You do not need to set or change anything on the next screen that comes up.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_27-1776192492961.png" style="width: 416px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26059iB771640F93A77180/image-dimensions/416x592?v=v2" width="416" height="592" role="button" title="jim_thorstad_27-1776192492961.png" alt="jim_thorstad_27-1776192492961.png" /></span></P> <P><SPAN>Your Deploy to Staging job will run. The first time it runs, you will see a message asking for permission to access the staging variable group. Give it permission.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_28-1776192523578.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26060iD3EBE40A401D6767/image-size/large?v=v2&amp;px=999" role="button" title="jim_thorstad_28-1776192523578.png" alt="jim_thorstad_28-1776192523578.png" /></span></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_29-1776192550923.png" style="width: 668px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26061iBCDD286DAA16CEE8/image-dimensions/668x366?v=v2" width="668" height="366" role="button" title="jim_thorstad_29-1776192550923.png" alt="jim_thorstad_29-1776192550923.png" /></span></P> <P><SPAN>When the staging deployment finishes, go find your Job and run it.&nbsp;</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_30-1776192577136.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26062i522C3E10E553544B/image-size/large?v=v2&amp;px=999" role="button" title="jim_thorstad_30-1776192577136.png" alt="jim_thorstad_30-1776192577136.png" /></span></P> <P><SPAN>See that the data was added by the service principal. Depending on the part of the UI, you will either see the App Registration’s App ID or Name as shown below.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_31-1776192598096.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26063i4ED82A11443C2B23/image-size/large?v=v2&amp;px=999" role="button" title="jim_thorstad_31-1776192598096.png" alt="jim_thorstad_31-1776192598096.png" /></span></P> <P><SPAN>While the code deployment, job execution, and data writing to the table were all performed by the Azure App Registration’s SP, it was the Workspace Admin who enabled it in Workspace Settings, and the Unity Catalog owner who granted it editor rights on the schema.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_32-1776192625054.png" style="width: 678px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26064i7E2F6CAC3C070DE3/image-dimensions/678x201?v=v2" width="678" height="201" role="button" title="jim_thorstad_32-1776192625054.png" alt="jim_thorstad_32-1776192625054.png" /></span></P> <H2><SPAN>Deploying to Production</SPAN></H2> <P><SPAN>When you are ready, click the Review button and approve running the prod deployment stage.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_33-1776192653029.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26065i7464395574BDC4DF/image-size/large?v=v2&amp;px=999" role="button" title="jim_thorstad_33-1776192653029.png" alt="jim_thorstad_33-1776192653029.png" /></span></P> <P><SPAN>Your bundle should be deployed to the prod workspace. Find the job and run it, then check the prod catalog to see that it ran.</SPAN></P> <P><SPAN>Congratulations!&nbsp; <span class="lia-unicode-emoji" title=":party_popper:">🎉</span>.&nbsp; &nbsp; If you made it this far, you are now an Azure Databricks CI/CD champion.&nbsp;<span class="lia-unicode-emoji" title=":beaming_face_with_smiling_eyes:">😁</span></SPAN></P> <P><SPAN>Please share your thoughts about this article in the comments section. What concepts did you learn from this blog that were helpful? Is there anything you don’t understand or are unsure of? What aspects of the example shown do you think should be expanded on for an intro CI/CD best practice article? What other topics would you like to be expanded on in another Technical Blog?</SPAN></P> <H2><SPAN>Comparing Two Approaches to Workload Identity Federation: Azure AD-Brokered Service Connections vs. Databricks OIDC Integration</SPAN></H2> <P><SPAN>The approach taken for integrating Azure DevOps and Databricks in this blog uses App registration-backed Azure DevOps service connections. This is a well-established pattern for securely federating your pipeline workload between the two cloud services. But Databricks recently introduced an alternative OIDC-based approach. How do these compare, and which is the best option?</SPAN></P> <P><SPAN>Both approaches use secure, token-based protocols (OIDC and OAuth). The difference is the issuer. In the blog, I use the DevOps service connection feature, which means the DevOps pipeline (via the AzureCLI@2 task) obtains an Azure AD access token and passes it to the Databricks CLI as DATABRICKS_TOKEN. This works well when your pipeline also needs access to other Azure resources (Storage, Key Vault, etc.) through the same service principal.&nbsp;</SPAN></P> <P><SPAN>Databricks also supports a newer alternative: OAuth token federation (also called workload identity federation or OIDC). With this approach, the Azure DevOps pipeline authenticates directly to Databricks — no Azure AD intermediary, no service connections, and no explicit token retrieval step. You create a federation policy on the Databricks service principal that trusts your Azure DevOps organization's OIDC issuer, and the Databricks CLI handles the token exchange automatically using the pipeline's built-in System.AccessToken.</SPAN></P> <P><SPAN>This does result in a simpler pipeline (no <FONT face="andale mono,times" color="#3366FF">AzureCLI@2</FONT> task, no Azure service connections), but it only authenticates to Databricks — if you need access to other Azure resources, you'll still need the other service connection approach shown in this blog. To learn more, see <A href="https://learn.microsoft.com/en-us/azure/databricks/dev-tools/auth/provider-azure-devops" target="_blank" rel="noopener">https://learn.microsoft.com/en-us/azure/databricks/dev-tools/auth/provider-azure-devops</A> in the Databricks documentation.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_1-1776277670883.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26113iBCF0FEFBC8F723D2/image-size/large?v=v2&amp;px=999" role="button" title="jim_thorstad_1-1776277670883.png" alt="jim_thorstad_1-1776277670883.png" /></span></P> <TABLE> <TBODY> <TR> <TD> <P><STRONG>Aspect</STRONG></P> </TD> <TD> <P><STRONG>Approach 1: </STRONG><STRONG><BR /></STRONG><STRONG>Azure AD-Brokered Authentication (This Blog)</STRONG></P> </TD> <TD> <P><STRONG>Approach 2: </STRONG><STRONG><BR /></STRONG><STRONG>Databricks-Native OIDC Federation (Alternate Option)</STRONG></P> </TD> </TR> <TR> <TD> <P><SPAN>Doc link</SPAN></P> </TD> <TD> <P><A href="https://learn.microsoft.com/en-us/azure/devops/pipelines/release/configure-workload-identity?view=azure-devops&amp;tabs=managed-identity" target="_blank" rel="noopener"><SPAN>Connect to Azure using App registration with workload identity federation</SPAN></A></P> </TD> <TD> <P><A href="https://learn.microsoft.com/en-us/azure/databricks/dev-tools/auth/provider-azure-devops" target="_blank" rel="noopener"><SPAN>Enable workload identity federation for Azure DevOps Pipelines</SPAN></A></P> </TD> </TR> <TR> <TD> <P><SPAN>Maturity</SPAN></P> </TD> <TD> <P><SPAN>Well-established pattern</SPAN></P> </TD> <TD> <P><SPAN>Newer (Databricks OAuth federation)</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>Protocols used</SPAN></P> </TD> <TD> <P><SPAN>OIDC, OAuth 2.0, bearer token</SPAN></P> </TD> <TD> <P><SPAN>OIDC → Databricks OAuth 2.0</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>Auth intermediary</SPAN></P> </TD> <TD> <P><SPAN>Azure AD (token broker)</SPAN></P> </TD> <TD> <P><SPAN>None (direct OIDC)</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>Token acquisition</SPAN></P> </TD> <TD> <P><SPAN>Explicit&nbsp; </SPAN><FONT face="andale mono,times" color="#3366FF"><SPAN>az account get-access-token</SPAN></FONT></P> </TD> <TD> <P><SPAN>Automatic by Databricks SDK/CLI</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>Azure CLI in pipeline</SPAN></P> </TD> <TD> <P><SPAN>Required</SPAN></P> </TD> <TD> <P><SPAN>Not needed</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>Azure service connections</SPAN></P> </TD> <TD> <P><SPAN>Required (one per env)</SPAN></P> </TD> <TD> <P><SPAN>Not needed</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>Federation policy</SPAN></P> </TD> <TD> <P><SPAN>On Azure AD (trusts Azure DevOps)</SPAN></P> </TD> <TD> <P><SPAN>On Databricks SP (trusts Azure DevOps)</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>Complexity</SPAN></P> </TD> <TD> <P><SPAN>Two hops; 8 pipeline steps per stage</SPAN></P> </TD> <TD> <P><SPAN>One hop; 2-3 pipeline steps per stage</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>Azure resource access</SPAN></P> </TD> <TD> <P><SPAN>The same SP can access other Azure resources (key vault, etc.).</SPAN></P> </TD> <TD> <P><SPAN>Only authenticates to Databricks</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>Better when</SPAN></P> </TD> <TD> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>The pipeline also needs to interact with other Azure resources (storage, Key Vault, etc.) using the same service principal</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Your org already has Azure AD service connections established and governed</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>You want the well-trodden, battle-tested path that most Azure shops already understand</SPAN></LI> </UL> </TD> <TD> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>The pipeline only needs Databricks access</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>You want the simplest possible pipeline YAML</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>You want to eliminate Azure AD as a dependency/intermediary</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>You're on a non-Azure cloud but still using Azure DevOps</SPAN> <SPAN>(the OIDC issuer is Azure DevOps itself, not Azure AD)</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>You want the Databricks SDK to handle the token lifecycle automatically</SPAN></LI> </UL> </TD> </TR> <TR> <TD> <P><SPAN>What’s different</SPAN></P> </TD> <TD>--</TD> <TD> <P data-unlink="true"><SPAN>Skips Azure AD entirely. A federation policy is created directly on the Databricks service principal that trusts the Azure DevOps OIDC issuer (</SPAN><FONT face="andale mono,times" color="#3366FF"><SPAN>https://vstoken.dev.azure.com/&lt;org_id&gt;</SPAN></FONT><SPAN>). The pipeline YAML is</SPAN></P> <P><SPAN>&nbsp;&nbsp;&nbsp;Minimal:<BR /></SPAN></P> <span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jim_thorstad_0-1776277460580.png" style="width: 463px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26112iB58A4B23E9344CE9/image-dimensions/463x139?v=v2" width="463" height="139" role="button" title="jim_thorstad_0-1776277460580.png" alt="jim_thorstad_0-1776277460580.png" /></span></TD> </TR> <TR> <TD> <P><SPAN>Requires</SPAN></P> </TD> <TD>--</TD> <TD> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>A Databricks service principal with a federation policy configured via </SPAN><SPAN><BR /></SPAN><FONT face="andale mono,times" color="#3366FF"><SPAN>databricks account service-principal-federation-policy create</SPAN></FONT></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>&nbsp;&nbsp;- Three environment variables + </SPAN><FONT face="andale mono,times" color="#3366FF"><SPAN>$(System.AccessToken)</SPAN></FONT></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>&nbsp;&nbsp;- No Azure AD, no Azure service connections, no </SPAN><FONT face="andale mono,times" color="#3366FF"><SPAN>AzureCLI@2</SPAN></FONT><SPAN> task</SPAN></LI> </UL> </TD> </TR> </TBODY> </TABLE> Thu, 16 Apr 2026 09:23:21 GMT jim_thorstad 2026-04-16T09:23:21Z https://community.databricks.com/t5/technical-blog/azure-devops-with-databricks-dabs-step-by-step-integration-guide/ba-p/141451 <P><SPAN>Establishing a trusted </SPAN><I>Continuous Integration/Continuous Deployment (CI/CD)</I> <SPAN>process is crucial for effectively managing the lifecycle of your data and AI workloads in Azure Databricks. However, with numerous setup options and a constantly evolving platform, the big question is, "Where do I even start?" This comprehensive blog cuts through the complexity to outline a </SPAN><A href="https://docs.databricks.com/aws/en/dev-tools/ci-cd/best-practices"><STRONG>recommended CI/CD approach</STRONG></A> <SPAN>for Azure Databricks.&nbsp;</SPAN></P> Thu, 16 Apr 2026 09:23:21 GMT https://community.databricks.com/t5/technical-blog/azure-devops-with-databricks-dabs-step-by-step-integration-guide/ba-p/141451 jim_thorstad 2026-04-16T09:23:21Z https://community.databricks.com/t5/technical-blog/azure-devops-with-databricks-dabs-step-by-step-integration-guide/bc-p/155001#M1005 <P>Amazing work on this&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/38005">@jim_thorstad</a>&nbsp;- exactly what I needed to get going!&nbsp; What an epic!</P> Tue, 21 Apr 2026 00:21:30 GMT https://community.databricks.com/t5/technical-blog/azure-devops-with-databricks-dabs-step-by-step-integration-guide/bc-p/155001#M1005 SamGreene 2026-04-21T00:21:30Z https://community.databricks.com/t5/technical-blog/azure-devops-with-databricks-dabs-step-by-step-integration-guide/bc-p/155073#M1009 <P>A M A Z I N G with all details..</P> Tue, 21 Apr 2026 11:45:48 GMT https://community.databricks.com/t5/technical-blog/azure-devops-with-databricks-dabs-step-by-step-integration-guide/bc-p/155073#M1009 h_h_ak 2026-04-21T11:45:48Z https://community.databricks.com/t5/technical-blog/azure-devops-with-databricks-dabs-step-by-step-integration-guide/bc-p/163794#M1129 <P>Amazing content! Thanks a lot!</P><P>&nbsp;</P> Wed, 22 Jul 2026 17:22:05 GMT https://community.databricks.com/t5/technical-blog/azure-devops-with-databricks-dabs-step-by-step-integration-guide/bc-p/163794#M1129 mrrickyno 2026-07-22T17:22:05Z