topic Re: Yml file replacing job cluster with all-purpose cluster when running a workflow in Data Engineering

Yml file replacing job cluster with all-purpose cluster when running a workflow

dataengutility — Mon, 10 Jun 2024 16:51:30 GMT

Hi all,

I have been having some trouble running a workflow that consists of 3 tasks that run sequentially. Task1 runs on an all-purpose cluster and kicks off Task2 that needs to run on a job cluster. Task2 kicks off Task3 which also uses a job cluster.

We have identified that Task2 is running on an all-purpose cluster instead of a job cluster despite configuring the task to run using a job cluster in the yaml file for the asset bundle. This task is dependent on another task which does use the all-purpose cluster as specified in the yaml file. We tried modifying the yaml file but when running a databricks bundle validate, it looks like the task is being overwritten to use the all-purpose cluster despite explicitly indicating it to use the job cluster. Renaming the task names is being picked up by the validate command.

Here is a snippet of the yaml file:

tasks: - task_key: Task1 existing_cluster_id: all-purpose-cluster-id notebook_task: notebook_path: ../src/Task1.py base_parameters: catalog: ${var.catalog} target: ${var.target} - task_key: Task2 job_cluster_key: job-cluster depends_on: - task_key: Task1 notebook_task: notebook_path: ../src/Task2.py base_parameters: catalog: ${var.catalog} target: ${var.target} - task_key: Task3 job_cluster_key: job-cluster depends_on: - task_key: Task2 notebook_task: notebook_path: ../src/Task3.py base_parameters: catalog: ${var.catalog} target: ${var.target}

After running databricks bundle validate, this is the output:

"tasks": [ { "existing_cluster_id": "all-purpose-cluster-id", "notebook_task": { "base_parameters": { "catalog": "catalog", "target": "target" }, "notebook_path": "/Users/user/.bundle/folder/dev/files/src/Task1" }, "task_key": "Task1" }, { "depends_on": [ { "task_key": "Task1" } ], "existing_cluster_id": "all-purpose-cluster-id", "notebook_task": { "base_parameters": { "catalog": "catalog", "target": "target" }, "notebook_path": "/Users/user/.bundle/folder/dev/files/src/Task2" }, "task_key": "Task2" }, { "depends_on": [ { "task_key": "Task2" } ], "existing_cluster_id": "all-purpose-cluster-id", "notebook_task": { "base_parameters": { "catalog": "catalog", "target": "target" }, "notebook_path": "/Users/user/.bundle/folder/dev/files/src/Task3" }, "task_key": "Task3" } ]

As you can see, the all-purpose cluster id is replacing the job-cluster key for Task2 and Task3. The strangest part of all of this is that I'm the only one on the team that is experiencing this issue. Everyone else seems to be able to run the workflow without any issues. Any ideas on how to resolve this issue?

Thank you in advanced!

Re: Yml file replacing job cluster with all-purpose cluster when running a workflow

jacovangelder — Tue, 11 Jun 2024 13:35:37 GMT

I don't know if you've cut off your yaml snippet, but your snippet doesn't show your job cluster with key job-cluster. Just to validate, your job cluster is also defined in your workflow yaml?

Edit: Looking it it again and knowing the defaults, it looks like you're pointing to job_cluster_key "job-cluster". The default is job_cluster (with an underscore instead of a regular hyphen). Could this be your issue?

Re: Yml file replacing job cluster with all-purpose cluster when running a workflow

dataengutility — Tue, 11 Jun 2024 18:44:04 GMT

Hi, thank you for your response! Yes, I did cut off the yaml snippet to the problem area since the entire yaml is quite a large file. We do define the job-cluster in the workflow yaml. "job-cluster" is just a pseudonym. Sorry for the confusing snippet. Here is the portion where the job cluster is defined:

job_clusters: - job_cluster_key: job-cluster new_cluster: spark_version: 12.2.x-scala2.12 node_type_id: m5d.4xlarge driver_node_type_id: m5d.4xlarge data_security_mode: SINGLE_USER runtime_engine: PHOTON autoscale: min_workers: 2 max_workers: 12 aws_attributes: instance_profile_arn: [removed] zone_id: auto first_on_demand: 1

Re: Yml file replacing job cluster with all-purpose cluster when running a workflow

jacovangelder — Wed, 12 Jun 2024 07:15:12 GMT

That should work just fine. Just tested it on my end. As long as your job_cluster_key value matches with the one in your task, it should work.

Perhaps you can try to throw away your bundle folder (and perhaps your workflows too) in your workspace and then deploy again. Do keep in mind that run history is purged because of this. It could be that the Terraform state is somehow messed up because of previous faulty deployments.

Re: Yml file replacing job cluster with all-purpose cluster when running a workflow

dataengutility — Mon, 17 Jun 2024 16:57:13 GMT

My issue is resolved. I had to upgrade my CLI version from v0.215 to v0.221 and everything works fine now. Thank you for your help!

Re: Yml file replacing job cluster with all-purpose cluster when running a workflow

vsharma — Sat, 21 Feb 2026 13:24:53 GMT

I am facing same issue , i see 2 clusters one ALL purpose which is default and job compute, in my YML I have configured to use only job compute, not sure how All Purpose is getting referenced ? I do not have any CLI issue as i am using latest ,