topic Re: How to Dynamically Retrieve Serverless Cluster ID for Databricks Job Configuration? in Data Engineering

How to Dynamically Retrieve Serverless Cluster ID for Databricks Job Configuration?

dunno — Thu, 09 Jan 2025 04:27:46 GMT

I am working on deploying a Databricks job to the production environment using a PowerShell script in Azure DevOps release pipeline. The task requires to update the job configuration JSON file to set the job's compute to serverless. For this, I need to set the `existing_cluster_id` field in the job config. Is there a way to programmatically retrieve the dynamic serverless cluster ID so that I can use it to update the `existing_cluster_id` field in the JSON job configuration?Here's the PowerShell code I am currently working with:

$serverlessId = "xxxx-xxxxxx-xxxxxxxx-xxx"
$workspaceBasePath = "/path/toWorkspace/"
$JobPath = "$(ARTIFACT_PATH)\DatabricksBundle\jobs\job-settings.json"

function Import-Job {

param (

        [Parameter(Mandatory = $true)]
        [string]$jsonPath
    )
    if (-not (Test-Path $jsonPath)) {
        Write-Output "Error: The job settings file does not exist at the path: $jsonPath"
        throw "File not found"
    }
    try {
    $jsonContent = Get-Content -Path $jsonPath | ConvertFrom-Json
    $jsonContent.PSObject.Properties.Remove("existing_cluster_id"
    $jsonContent | Add-Member -MemberType NoteProperty -Name "existing_cluster_id" -Value $(serverlessId)
    $jsonPayload = $jsonContent | ConvertTo-Json -Depth 10
    Write-Output "job-config-settings content: $jsonPayload"
        $response = Invoke-RestMethod -Uri "$databricksUrl/api/2.1/jobs/create" -Method Post -Headers $headers -Body $jsonPayload -ContentType "application/json"
        Write-Output "Job created successfully: $($response.job_id)"
    } catch {
        Write-Output "Error creating job: $_"
    }
}

Import-Job -jsonPath $JobPath

Any suggestions or examples on how to get the serverless cluster ID in PowerShell would be greatly appreciated!

Re: How to Dynamically Retrieve Serverless Cluster ID for Databricks Job Configuration?

Walter_C — Thu, 09 Jan 2025 10:03:57 GMT

Is this field completely required? Based on API job deployment, by not setting any details for cluster on the job the system will set it automatically as Serverless, this as for jobs the serverless will not be the same on each run.

Re: How to Dynamically Retrieve Serverless Cluster ID for Databricks Job Configuration?

dunno — Wed, 15 Jan 2025 05:58:21 GMT

Thank you. It worked with multi-task jobs. I was trying with a single task job, and it looks like it requires this id, right? Is there a way to deploy single task jobs in the serverless compute

Re: How to Dynamically Retrieve Serverless Cluster ID for Databricks Job Configuration?

Walter_C — Wed, 15 Jan 2025 14:31:59 GMT

As per docs, the single task mode has been deprecated, so you will need to use the multi task method as of now https://docs.databricks.com/api/workspace/jobs/create#format

Re: How to Dynamically Retrieve Serverless Cluster ID for Databricks Job Configuration?

dunno — Wed, 15 Jan 2025 15:32:15 GMT

Got it! Thanks for the info.

Re: How to Dynamically Retrieve Serverless Cluster ID for Databricks Job Configuration?

Walter_C — Wed, 15 Jan 2025 16:14:58 GMT

sure, happy to assist, if any of my responses was able to help you would really appreciate if you can accept it as a solution