filipniziol
Esteemed Contributor

Hi @oakhill ,

You can specify you job cluster configuration in your variables:

variables:
  small_cluster_id:
    description: "The small cluster with 2 workers used by the jobs"
    type: complex
    default:
      spark_version: "15.4.x-scala2.12"
      node_type_id: "Standard_D4ds_v5"
      num_workers: 2

Now you can specify using this cluster in your jobs:

resources:
  jobs:
    my_job:
      name: my_job

      job_clusters:
        - job_cluster_key: small_cluster
          new_cluster: ${var.small_cluster_id}