I am working on deploying a Databricks job to the production environment using a PowerShell script in Azure DevOps release pipeline. The task requires to update the job configuration JSON file to set the job's compute to serverless. For this, I need to set the `existing_cluster_id` field in the job config. Is there a way to programmatically retrieve the dynamic serverless cluster ID so that I can use it to update the `existing_cluster_id` field in the JSON job configuration?Here's the PowerShell code I am currently working with:
$serverlessId = "xxxx-xxxxxx-xxxxxxxx-xxx"
$workspaceBasePath = "/path/toWorkspace/"
$JobPath = "$(ARTIFACT_PATH)\DatabricksBundle\jobs\job-settings.json"
function Import-Job {
param (
[Parameter(Mandatory = $true)]
[string]$jsonPath
)
if (-not (Test-Path $jsonPath)) {
Write-Output "Error: The job settings file does not exist at the path: $jsonPath"
throw "File not found"
}
try {
$jsonContent = Get-Content -Path $jsonPath | ConvertFrom-Json
$jsonContent.PSObject.Properties.Remove("existing_cluster_id"
$jsonContent | Add-Member -MemberType NoteProperty -Name "existing_cluster_id" -Value $(serverlessId)
$jsonPayload = $jsonContent | ConvertTo-Json -Depth 10
Write-Output "job-config-settings content: $jsonPayload"
$response = Invoke-RestMethod -Uri "$databricksUrl/api/2.1/jobs/create" -Method Post -Headers $headers -Body $jsonPayload -ContentType "application/json"
Write-Output "Job created successfully: $($response.job_id)"
} catch {
Write-Output "Error creating job: $_"
}
}
Import-Job -jsonPath $JobPath
Any suggestions or examples on how to get the serverless cluster ID in PowerShell would be greatly appreciated!
