Hi everyone,
I am working on setting up a complete end-to-end CI/CD process for my Databricks environment using Azure DevOps. So far, I have developed a build pipeline to create a Databricks artifact (DAB).
Now, I need to create a release pipeline to deploy this artifact into production. My plan is to use the artifact from the build pipeline and the Databricks REST API to push it into production.
Questions:
- Will this approach publish workflows and notebooks into production exactly as they are in the development environment?
- Are there any best practices or recommendations for structuring the release pipeline?
I am new to this and would appreciate any suggestions.
Below is the code I’m currently using in the release pipeline.
Release Pipeline Code:
# Define Databricks variables
$databricksUrl = "<Databricks-URL>" # Replace with your Databricks instance URL
$accessToken = "<Access-Token>" # Replace with your secure token
# Define headers for Databricks REST API
$headers = @{
"Authorization" = "Bearer $accessToken"
}
# Paths inside the Databricks workspace
$workspaceBasePath = ""
$notebookPath = ""
$jobPath = ""
# Function to create directories in Databricks
function Create-Directory {
param ([string]$directoryPath)
$createDirUri = "$databricksUrl/api/2.0/workspace/mkdirs"
$body = @{ "path" = $directoryPath }
try {
Invoke-RestMethod -Method POST -Uri $createDirUri -Headers $headers -Body ($body | ConvertTo-Json -Depth 10) -ContentType "application/json"
Write-Output "Directory '$directoryPath' created successfully in Databricks."
} catch {
if ($_.Exception.Response.StatusCode -ne 400) {
Write-Error "Failed to create directory '$directoryPath': $_"
}
}
}
# Additional functions (Delete-File, Import-Notebook, Import-Job) are implemented similarly to handle file deletions and imports.
# Example pipeline steps:
Create-Directory -directoryPath "$workspaceBasePath/notebooks"
Create-Directory -directoryPath "$workspaceBasePath/jobs"
Delete-File -filePath "$workspaceBasePath/notebooks/Contingent_Employee_Report"
Delete-File -filePath "$workspaceBasePath/jobs/job-config.json"
Import-Notebook -notebookPath $notebookPath -workspacePath "$workspaceBasePath/notebooks/Contingent_Employee_Report"
Import-Job -jobConfigJsonPath $jobPathThank you in advance for your time and suggestions!
