Different settings per target with Asset bundles

[Mathias]

When generating the standard setup with databricks bundle init we will get databricks.yml that references resources/*. The targets are set in the databricks.yml and the resources (pipelines and jobs) are set in different files.

I have dlt pipelines that I want to run continuously in the production workspace but, to save cost, want to run triggered in the dev workspace. My pipelines in Azure Devops deploy the code first to dev and then to prod using the databricks bundle deploy -t xxx.

Is there a best practice for how to implement the differences?

I tried adding an if statement but it doesn't seem to be working

      ${{ if eq(${bundle.environment}, 'dev') }}:

        continuous: false

      ${{ elseif eq(${bundle.environment}, 'prod' ) }}:

        continuous: true

[Kaniz_Fatma]

Hi @Mathias , 

To configure the continuous flag for your Delta Live Tables pipelines differently for dev and prod workspaces, it's best to use separate YAML files for each environment. Here's the approach:

1. Create Separate Pipeline YAML Files: Generate distinct YAML files (e.g., dlt_pipeline_dev.yml and dlt_pipeline_prod.yml) for your Delta Live Tables pipelines in each environment. Set the continuous flag accordingly in these files.

   dlt_pipeline_dev.yml:

   apiVersion: 1.3
   name: my_dlt_pipeline_dev
   pipelines:
     - name: my_pipeline_dev
       continuous: false
       source_path: /mnt/dev/source/path/
       destination_path: /mnt/dev/destination/path/
       schema_metaline: "col1 int, col2 string"
   

   dlt_pipeline_prod.yml:

   apiVersion: 1.3
   name: my_dlt_pipeline_prod
   pipelines:
     - name: my_pipeline_prod
       continuous: true
       source_path: /mnt/prod/source/path/
       destination_path: /mnt/prod/destination/path/
       schema_metaline: "col1 int, col2 string"
   

   2. Deploy the Appropriate Pipeline File: When deploying your Delta Live Tables pipelines to each environment, select the relevant pipeline file using the -f option. For example:

   For Dev:

   databricks bundle deploy -t dev -f dlt_pipeline_dev.yml

   For Prod:

   databricks bundle deploy -t prod -f dlt_pipeline_prod.yml

2. By using separate YAML files for each environment, you can ensure that the continuous flag is correctly set for each pipeline, tailored to the specific requirements of that environment.

[pietern]

@Kaniz_Fatma The YAML snippets you included aren't valid bundle configurations. The `-f` flag also doesn't exist.

DABs are always deployed in their entirety, not selectively.

The configuration syntax reference can be found here: https://docs.databricks.com/en/dev-tools/bundles/settings.html

[Mathias]

In our case we are using build pipelines in Azure Devops do deploy the solution, which consist or multiple different jobs and dlt pipelines. We are using Databricks bundles as created by databricks bundle init, which creates one databricks.yml and a Resources -folder, where the individual pipeline specifications reside.

To give context, our build pipeline looks like this:

 stage: Release_To_Dev

  variables:

    - group: databricks-dev-env

  jobs:

  - template: deploy-steps.yml

    parameters:

      environment: dev

- stage: Release_To_Production

  dependsOn: Release_To_Dev

  variables:

    - group: databricks-prod-env

  condition: and(succeeded('Release_To_Dev'), eq(variables['Build.SourceBranch'], 'refs/heads/main'))

  jobs:

  - template: deploy-steps.yml

    parameters:

      environment: prod

The deploy-steps.yml looks like this:

parameters:

- name: Environment

jobs:

- job: BuildAndRun

    - script: |

        curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh

      displayName: 'Install Databricks CLI'

    - script: |

        /usr/local/bin/databricks bundle deploy --target ${{ parameters.environment }}

      env:

        DATABRICKS_HOST: $(databricks_host)

        ARM_TENANT_ID: $(arm_tenant_id)

        ARM_CLIENT_ID: $(arm_client_id)

        ARM_CLIENT_SECRET: $(arm_client_secret)

      displayName: 'Deploy the pipeline, jobs and code files defined in the bundle'

Any recommendations on how to specify different settings per environment in this case?

[pietern]

It is possible to use target overrides to customize resources based on the target you're deploying to.

Documentation can be found here: https://docs.databricks.com/en/dev-tools/bundles/settings.html#targets

An example of this pattern can be found here: https://github.com/databricks/cli/blob/main/bundle/tests/override_pipeline_cluster/databricks.yml

Instead of the `clusters` or `name` field, you would include the `continuous` field.

[JacekLaskowski]

That's exactly my words! I'd not be surprised if this were the author of DAB judging by the nickname (https://github.com/databricks/cli/commits?author=pietern) 😉

[137292]

-f is unknown shorthand flag for databricks bundle deploy. Any workaround on how to deploy different jobs with different targets?

[MohsenJ]

I also like to know the solution to this problem