topic Re: Different settings per target with Asset bundles in Administration & Architecture

Different settings per target with Asset bundles

Mathias — Thu, 26 Oct 2023 14:06:06 GMT

When generating the standard setup with databricks bundle init we will get databricks.yml that references resources/*. The targets are set in the databricks.yml and the resources (pipelines and jobs) are set in different files.

I have dlt pipelines that I want to run continuously in the production workspace but, to save cost, want to run triggered in the dev workspace. My pipelines in Azure Devops deploy the code first to dev and then to prod using the databricks bundle deploy -t xxx.

Is there a best practice for how to implement the differences?

I tried adding an if statement but it doesn't seem to be working

${{ if eq(${bundle.environment}, 'dev') }}:

continuous: false

${{ elseif eq(${bundle.environment}, 'prod' ) }}:

continuous: true

Re: Different settings per target with Asset bundles

Mathias — Thu, 02 Nov 2023 14:46:32 GMT

In our case we are using build pipelines in Azure Devops do deploy the solution, which consist or multiple different jobs and dlt pipelines. We are using Databricks bundles as created by databricks bundle init, which creates one databricks.yml and a Resources -folder, where the individual pipeline specifications reside.

To give context, our build pipeline looks like this:

stage: Release_To_Dev

variables:

- group: databricks-dev-env

jobs:

- template: deploy-steps.yml

parameters:

environment: dev

- stage: Release_To_Production

dependsOn: Release_To_Dev

variables:

- group: databricks-prod-env

condition: and(succeeded('Release_To_Dev'), eq(variables['Build.SourceBranch'], 'refs/heads/main'))

jobs:

- template: deploy-steps.yml

parameters:

environment: prod

The deploy-steps.yml looks like this:

parameters:

- name: Environment

jobs:

- job: BuildAndRun

- script: |

curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh

displayName: 'Install Databricks CLI'

- script: |

/usr/local/bin/databricks bundle deploy --target ${{ parameters.environment }}

env:

DATABRICKS_HOST: $(databricks_host)

ARM_TENANT_ID: $(arm_tenant_id)

ARM_CLIENT_ID: $(arm_client_id)

ARM_CLIENT_SECRET: $(arm_client_secret)

displayName: 'Deploy the pipeline, jobs and code files defined in the bundle'

Any recommendations on how to specify different settings per environment in this case?

Re: Different settings per target with Asset bundles

137292 — Fri, 23 Feb 2024 19:19:25 GMT

-f is unknown shorthand flag for databricks bundle deploy. Any workaround on how to deploy different jobs with different targets?

Re: Different settings per target with Asset bundles

pietern — Fri, 01 Mar 2024 15:18:13 GMT

@Retired_mod The YAML snippets you included aren't valid bundle configurations. The `-f` flag also doesn't exist.

DABs are always deployed in their entirety, not selectively.

The configuration syntax reference can be found here: https://docs.databricks.com/en/dev-tools/bundles/settings.html

Re: Different settings per target with Asset bundles

pietern — Fri, 01 Mar 2024 15:22:13 GMT

It is possible to use target overrides to customize resources based on the target you're deploying to.

Documentation can be found here: https://docs.databricks.com/en/dev-tools/bundles/settings.html#targets

An example of this pattern can be found here: https://github.com/databricks/cli/blob/main/bundle/tests/override_pipeline_cluster/databricks.yml

Instead of the `clusters` or `name` field, you would include the `continuous` field.

Re: Different settings per target with Asset bundles

MohsenJ — Mon, 27 May 2024 14:28:25 GMT

I also like to know the solution to this problem

Re: Different settings per target with Asset bundles

JacekLaskowski — Wed, 17 Jul 2024 15:09:52 GMT

That's exactly my words! I'd not be surprised if this were the author of DAB judging by the nickname (https://github.com/databricks/cli/commits?author=pietern) 😉