cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Different settings per target with Asset bundles

Mathias
New Contributor II

When generating the standard setup with databricks bundle init we will get databricks.yml that references resources/*. The targets are set in the databricks.yml and the resources (pipelines and jobs) are set in different files.

I have dlt pipelines that I want to run continuously in the production workspace but, to save cost, want to run triggered in the dev workspace. My pipelines in Azure Devops deploy the code first to dev and then to prod using the databricks bundle deploy -t xxx.

Is there a best practice for how to implement the differences?

I tried adding an if statement but it doesn't seem to be working

      ${{ if eq(${bundle.environment}, 'dev') }}:
        continuous: false
      ${{ elseif eq(${bundle.environment}, 'prod' ) }}:
        continuous: true
 
7 REPLIES 7

Kaniz_Fatma
Community Manager
Community Manager

Hi @Mathias , 

To configure the continuous flag for your Delta Live Tables pipelines differently for dev and prod workspaces, it's best to use separate YAML files for each environment. Here's the approach:

  1. Create Separate Pipeline YAML Files: Generate distinct YAML files (e.g., dlt_pipeline_dev.yml and dlt_pipeline_prod.yml) for your Delta Live Tables pipelines in each environment. Set the continuous flag accordingly in these files.

    dlt_pipeline_dev.yml:

    apiVersion: 1.3
    name: my_dlt_pipeline_dev
    pipelines:
      - name: my_pipeline_dev
        continuous: false
        source_path: /mnt/dev/source/path/
        destination_path: /mnt/dev/destination/path/
        schema_metaline: "col1 int, col2 string"
    
    dlt_pipeline_prod.yml:
    apiVersion: 1.3
    name: my_dlt_pipeline_prod
    pipelines:
      - name: my_pipeline_prod
        continuous: true
        source_path: /mnt/prod/source/path/
        destination_path: /mnt/prod/destination/path/
        schema_metaline: "col1 int, col2 string"
    

    2. Deploy the Appropriate Pipeline File: When deploying your Delta Live Tables pipelines to each environment, select the relevant pipeline file using the -f option. For example:

    For Dev:

    databricks bundle deploy -t dev -f dlt_pipeline_dev.yml

    For Prod:

    databricks bundle deploy -t prod -f dlt_pipeline_prod.yml
  2. By using separate YAML files for each environment, you can ensure that the continuous flag is correctly set for each pipeline, tailored to the specific requirements of that environment.

pietern
New Contributor III

@Kaniz_Fatma The YAML snippets you included aren't valid bundle configurations. The `-f` flag also doesn't exist.

DABs are always deployed in their entirety, not selectively.

The configuration syntax reference can be found here: https://docs.databricks.com/en/dev-tools/bundles/settings.html

Mathias
New Contributor II

In our case we are using build pipelines in Azure Devops do deploy the solution, which consist or multiple different jobs and dlt pipelines. We are using Databricks bundles as created by databricks bundle init, which creates one databricks.yml and a Resources -folder, where the individual pipeline specifications reside.

To give context, our build pipeline looks like this:

 stage: Release_To_Dev
  variables:
    - group: databricks-dev-env
  jobs:
  - template: deploy-steps.yml
    parameters:
      environment: dev

- stage: Release_To_Production
  dependsOn: Release_To_Dev
  variables:
    - group: databricks-prod-env
  condition: and(succeeded('Release_To_Dev'), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
  jobs:
  - template: deploy-steps.yml
    parameters:
      environment: prod

The deploy-steps.yml looks like this:

parameters:
- name: Environment

jobs:
- job: BuildAndRun
    - script: |
      displayName: 'Install Databricks CLI'

 
    - script: |
        /usr/local/bin/databricks bundle deploy --target ${{ parameters.environment }}
      env:
        DATABRICKS_HOST: $(databricks_host)
        ARM_TENANT_ID: $(arm_tenant_id)
        ARM_CLIENT_ID: $(arm_client_id)
        ARM_CLIENT_SECRET: $(arm_client_secret)
      displayName: 'Deploy the pipeline, jobs and code files defined in the bundle'

Any recommendations on how to specify different settings per environment in this case?

pietern
New Contributor III

It is possible to use target overrides to customize resources based on the target you're deploying to.

Documentation can be found here: https://docs.databricks.com/en/dev-tools/bundles/settings.html#targets

An example of this pattern can be found here: https://github.com/databricks/cli/blob/main/bundle/tests/override_pipeline_cluster/databricks.yml

Instead of the `clusters` or `name` field, you would include the `continuous` field.

That's exactly my words! I'd not be surprised if this were the author of DAB judging by the nickname (https://github.com/databricks/cli/commits?author=pietern😉

137292
New Contributor II

-f is unknown shorthand flag for databricks bundle deploy. Any workaround on how to deploy different jobs with different targets?

MohsenJ
Contributor

I also like to know the solution to this problem

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group