cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Variables in databricks.yml "include:" - Asset Bundles

nickneoners
New Contributor II

HI,

We've got an app that we deploy to multiple customers workspaces. 

We're looking to transition to asset bundles. We would like to structure our resources like:

 

 

-src/
-resources/
|-- customer_1/
   |-- job_1
   |-- job_2
|-- customer_2/
   |-- job_3
   |-- job_4
|-- customer_3/
   |-- job_5
   |-- job_6
- databricks.yml

 

 

so each customer only gets their specific workflows.

In `databricks.yml` would love to do:

 

 

bundle:
  name: my_app

include: 
  - resources/${bundle.target}/*.yml

 

 

but unfortunately it seems you can't pass any variables into the `include` block.

Any ideas?

Thanks!

5 REPLIES 5

nickneoners
New Contributor II

Just to add to the above, we've got about 30-40 jobs per customer, so we don't want to define these jobs in the `databricks.yml`, which seems to be the only way to navigate around this issue

p4pratikjain
Contributor

Interesting use case!! Ideally having seperate bundle for each customer seems like a clean solution. But if you dont want that then - 
You can just include all the yaml files in databricks.yml with 

include: 
  - resources/*/*.yml

 Inside the yaml files handle different workspaces under different targets ? target is a top level node which is merged so you can have 'customer_1', 'customer_2' as different targets ?

This is not the intended use of targets but I guess this will solve the ask.

Pratik Jain

pietern
Databricks Employee
Databricks Employee

If the jobs you're defining per customer are completely different and don't share anything (e.g. some base configuration), then using targets for this purpose is not a great fit.

If you're looking to share files between these bundles (maybe some notebooks, maybe a wheel build, maybe some Python files, etc), then you could look into this example: https://github.com/databricks/bundle-examples/tree/main/knowledge_base/share_files_across_bundles

This enables you do define a separate and isolated bundle per customer, with a different deployment schedule for each one of them, but still share the same set of files between these deployments.

Breno_Ribeiro
New Contributor II

I have a similar use case. We have two different host for databricks, EU and NA. In some case we need to deploy a similar job in both hosts. To fix that, here how I did:
- Into job folder I created different job files, each one for one host. In aditional I created an empty job file, named job.yml

- src/
- jobs/
     |--- job_EU.yml (job configuration for EU dbx)
     |--- job_NA.yml (job configuration for NA dbx)
     |--- job.yml (empty file)

- In databricks.yml I include only the empty file:

include:
    - job.yml

- Finally, at workflow file, in each job for databricks commands I edit the empty file pasting the content of the specific job file, depending on the host I want to deploy and run. This step must be done in each job that calls a databricks bundle command. If they are at the same job, just needed once:

deploy:
   .
   .
   .
   steps:
      .
      .
      .
      - run: cat jobs/job_EU.yml > jobs/job.yml
      - run: databricks bundle deploy

 

 

Hi Breno,

If the jobs you're deploying are similar, it sounds like a good fit for target overrides. You'd define the core of the job just one, and specialize it on a per-target basis. For example, you could define a "eu" and "na" target, each pointing to their own workspace, and specialize the job in the target overrides section. You can find an example on how to do this for job clusters here: https://docs.databricks.com/en/dev-tools/bundles/cluster-override.html#example-2-conflicting-new-job... . All job properties can be overridden in a similar way.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group