topic Re: Using variables with Databricks Asset Bundles not working in Machine Learning

Using variables with Databricks Asset Bundles not working

NielsMH — Wed, 04 Sep 2024 13:20:10 GMT

Hi Everyone

I am using the default Databricks Asset Bundle. All is good except i can get stage dependt variables to work.

i have a bundle config file that looks like the below, where i have defined variables for each stage.

bundle: name: xxxxx include: - resources/*.yml targets: # The 'dev' target, for development purposes. This target is the default. dev: variables: env: description: target environment for fetching secrets from the relevant keyvault scope default: test # We use 'mode: development' to indicate this is a personal development copy: # - Deployed resources get prefixed with '[dev my_user_name]' # - Any job schedules and triggers are paused by default # - The 'development' mode is used for Delta Live Tables pipelines mode: development default: true workspace: host: adb-xxxxx.azuredatabricks.net # The 'prod' target, used for production deployment. prod: variables: env: description: target environment for fetching secrets from the relevant keyvault scope default: prod # We use 'mode: production' to indicate this is a production deployment. # Doing so enables strict verification of the settings below. mode: production workspace: host: adb-xxxxx.azuredatabricks.net # We always use /Users/xxx@xxx.dk for all resources to make sure we only have a single copy. # If this path results in an error, please make sure you have a recent version of the CLI installed. root_path: /Users/xxx@xxx.dk/.bundle/${bundle.name}/${bundle.target} run_as: # This runs as xxx@xxx.dk in production. We could also use a service principal here, # see https://docs.databricks.com/dev-tools/bundles/permissions.html. user_name: xxx@xxx.dk

however, when i run fx "databricks bundle deploy" i get the error "Error: variable env is not defined but is assigned a value"

i have tried to set the env var with --var="env=test" also

I try to call the env var from the base_parameters slot in following resource file .yml:

# The main job for xxxx. resources: jobs: xxxxx: name: xxxx schedule: # Run At 09:00:00am, on every Saturday, every month quartz_cron_expression: '0 0 9 ? * SAT *' timezone_id: Europe/Amsterdam email_notifications: on_failure: - xxxx tasks: - task_key: notebook_task job_cluster_key: job_cluster notebook_task: notebook_path: ../src/notebook.ipynb base_parameters: ENV: ${var.env} job_clusters: - job_cluster_key: job_cluster new_cluster: spark_version: 13.3.x-scala2.12 node_type_id: Standard_D3_v2 autoscale: min_workers: 1 max_workers: 1

help would be much appreciated.

kind regards

Niels

Re: Using variables with Databricks Asset Bundles not working

szymon_dybczak — Wed, 04 Sep 2024 13:51:20 GMT

Hi @NielsMH ,

I think there is small issue with your template. You missed variables mapping at the top level.

According to the documentation:

"The bundles settings file can contain one top-level variables mapping to specify variable settings to use."

Add env variable to the root level. Then the error that is saying that you didn't define variable should vanished.

And you can override top level variables with target mapping, as in your example 😉

https://learn.microsoft.com/en-us/azure/databricks/dev-tools/bundles/settings#targets

Re: Using variables with Databricks Asset Bundles not working

NielsMH — Thu, 05 Sep 2024 06:42:24 GMT

Hi Slash. That was indeed the problem. Thanks! 🙂

Re: Using variables with Databricks Asset Bundles not working

szymon_dybczak — Thu, 05 Sep 2024 07:05:16 GMT

Great, glad that it worked!

Re: Using variables with Databricks Asset Bundles not working

stupefyyy — Mon, 27 Jan 2025 17:39:36 GMT

yes this worked, you need to add the variables at both places i.e. the top level and set default values, and then add them again to the target as a mapping ( non default values).

bundle: # Required.
  name: string # Required.
  databricks_cli_version: string
  cluster_id: string
  git:
    origin_url: string
    branch: string

# These are for any custom variables for use throughout the bundle.
variables:
  <some-unique-variable-name>:
    description: string
    default: string or complex

targets:
  <some-unique-programmatic-identifier-for-this-target>:
    artifacts:
      # See the preceding "artifacts" syntax.
    bundle:
      # See the preceding "bundle" syntax.
    cluster_id: string
    default: true | false
    mode: development
    presets:
      <preset>: <value>
    resources:
      # See the preceding "resources" syntax.
    sync:
      # See the preceding "sync" syntax.
    variables:
      <preceding-unique-variable-name>: <non-default-value>
    workspace:
      # See the preceding "workspace" syntax.
    run_as:
      # See the preceding "run_as" syntax.