How do we create a job cluster in Databricks Asset Bundles for use across different jobs?

[oakhill]

When developing jobs on DABs, we use new_cluster to create a cluster for a particular job. I think it's a lot of lines and YAML when what I really need is a "small cluster" and "big cluster" to reference for certain kind of jobs. Tags would be on the job and get propagated to the cluster.

According to the dab-docs, we can create a cluster under resources: clusters: but surely we are not meant to use all-purpose clusters as a job cluster?

See the example here: https://docs.databricks.com/en/dev-tools/bundles/resources.html#cluster

[filipniziol]

Hi @oakhill ,

You can specify you job cluster configuration in your variables:

variables:
  small_cluster_id:
    description: "The small cluster with 2 workers used by the jobs"
    type: complex
    default:
      spark_version: "15.4.x-scala2.12"
      node_type_id: "Standard_D4ds_v5"
      num_workers: 2

Now you can specify using this cluster in your jobs:

resources:
  jobs:
    my_job:
      name: my_job

      job_clusters:
        - job_cluster_key: small_cluster
          new_cluster: ${var.small_cluster_id}