Hello,
Jobs are specific in Databricks; a job definition also contains the cluster definition because when you run a job, a new cluster is created based on the cluster specification you provided for the job, and it exists only until the job is completed. You can define a cluster on the job level or for individual tasks. You can use the same cluster within a job, so multiple tasks can be run on the same cluster.
If you want, you can share a cluster between jobs, BUT! it will be an All Purpose Cluster that costs 2x more DBUs. It is not recommended to use all-purpose clusters for jobs unless you have very specific needs.
Asset bundles are not very well documented yet (in public preview), so you can always refer to the API documentation:
In the documentation, there is an example where the script uses:
existing_cluster_idThis ID is ID of All Purpose Cluster that you can find in JSON definition of a cluster.
so in YAML it will be:
tasks:
- task_key: notebook_task
existing_cluster_id: Id of your existing Cluster
But as I mentioned, it is recommended to use Job Clusters. You can define multiple job clusters, for example, 2 clusters:
job_clusters:
- job_cluster_key: <some-unique-programmatic-identifier-for-this-key>
new_cluster:
# Cluster settings.
And use them WITHIN the job by assigning job_cluster_key to task specifications.
In this section of documentation you can see how you can do it:
