topic Re: Why does my job run on shared compute instead of job compute? in Data Engineering

Why does my job run on shared compute instead of job compute?

cm04 — Fri, 06 Sep 2024 17:24:59 GMT

I have configured a job using `databricks.yml`

```

resources:

jobs:

my_job:

name: my_job

tasks:

- task_key: create_feature_tables

job_cluster_key: my_job_cluster

spark_python_task:

python_file: ../src/create_feature_tables.py

- task_key: evaluate_model

job_cluster_key: my_job_cluster

spark_python_task:

python_file: ../src/evaluate_model.py

job_clusters:

- job_cluster_key: my_job_cluster

new_cluster:

policy_id: ${var.job_cluster_policy}

spark_version: 15.2.x-cpu-ml-scala2.12

node_type_id: i3.xlarge

aws_attributes:

first_on_demand: 1

autoscale:

min_workers: 1

max_workers: 12

```
For some reason, the job runs on shared compute, not my_job_compute as configured in the yml file:

How can I coax Databricks to use my_job_cluster, not shared compute?

szymon_dybczak — Mon, 09 Sep 2024 06:02:03 GMT

You can try to upgrade CLI to newest version. I've seen similiar issue before and upgrading CLI was a solution back then.

cm04 — Mon, 09 Sep 2024 13:09:04 GMT

Thanks! Updating CLI from v .224 to v0.228 seems to have resolved the issue.