How Can I update the DBR versions of all my jobs in one go?

brickster_2018 — Fri, 25 Jun 2021 13:08:59 GMT

I keep it a point to use the latest DBR versions for my workloads and mostly we leverage those new features. But I have 300 jobs on the Databricks workspace and updating the DBR versions for each job manually is difficult to do. Any quick hack

Re: How Can I update the DBR versions of all my jobs in one go?

brickster_2018 — Fri, 25 Jun 2021 13:10:27 GMT

Below code snippet can be helpful if you are using Databricks CLI

for jobid in `databricks jobs list | awk '{print $1}'`;
 do databricks jobs get --job-id $jobid | jq .settings > /tmp/jobs/$jobid.json; done
 
sed -i 's/"spark_version": ".*"/"spark_version": "6.4.x-scala2.11"/g' *
 
for jobid in `databricks jobs list | awk '{print $1}'`; do
databricks jobs reset --job-id $jobid --json-file /tmp/jobs/$jobid.json;
done

topic How Can I update the DBR versions of all my jobs in one go? in Data Engineering

How Can I update the DBR versions of all my jobs in one go?

Re: How Can I update the DBR versions of all my jobs in one go?