Run multiple jobs with different source code at the same time with Databricks asset bundles

curiousoctopus
New Contributor III

Hi,

I am migrating from dbx to databricks asset bundles. Previously with dbx I could work on different features in separate branches and launch jobs without issue of one job overwritting the other. Now with databricks asset bundles it seems like I can't since it's deploying/updating ONE job and running an instance of the latest.

This is what I have in my `databricks.yml` to deploy my job:

resources:
  jobs:
    <my-job>:
      name: my-job-${var.suffix}
      tasks:
        - ...

 I thought I could use a custom variable (here suffix) to create multiple jobs with the feature name as a suffix for example so that everyone working on different features could run their experiments. However it just changed the name of the job previously deployed. I also tried using the custom variable within the key <my-job> but it wasn't allowed.

 

So my question is how can I achieve this? Ultimately I want to be able to work on a different feature than my colleagues and not have to coordinate when I can launch my job to not overwrite theirs.

 

Thanks you.