- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-01-2024 04:50 AM - edited 04-01-2024 04:52 AM
Yup, totally agree with you it would be great if we have the ability to use include/exclude at that level. But anyway, as mentioned by Ossinova, adding your job 'test_job.yml' contents (resources mapping) into the target mapping with that job (or more than one) could solve your problem.
Check here about https://docs.databricks.com/en/dev-tools/bundles/settings.html#targets: "If a target mapping specifies a workspace, artifacts, or resources mapping, and a top-level workspace, artifacts, or resources mapping also exists, then any conflicting settings are overridden by the settings within the target.".
That means the new job (or whatever) resource in your case will be appended to the existing ones if you didn't introduce any conflict (make sure names are different).
Here is an example how I added a job to run only in test environment (and not in "dev", "staging" and "prod"):
# The name of the bundle. run `databricks bundle schema` to see the full bundle settings schema.
bundle:
name: mlops-stacks
variables:
experiment_name:
description: Experiment name for the model training.
default: /Users/${workspace.current_user.userName}/${bundle.target}-mlops-stacks-experiment
model_name:
description: Model name for the model training.
default: mlops-stacks-model
seperator:
description: useful seperator index by PR number for test workflows. Default is nothing for other envs
default: ""
include:
- ./assets/*.yml
# Deployment Target specific values for workspace
targets:
dev:
default: true
workspace:
host: https://********************.databricks.com
staging:
workspace:
host: https://********************.databricks.com
prod:
workspace:
host: https://********************.databricks.com
test:
workspace:
host: https://********************.databricks.com
# dedicated path to deploy files for test envs by PR
root_path: /Users/${workspace.current_user.userName}/.bundle/${bundle.name}/${bundle.target}${var.seperator}
variables:
# overwrite default experiment_name to have experiment by PR in test env
# (avoids "cannot create mlflow experiment: Node named '...-experiment' already exists")
experiment_name: /Users/${workspace.current_user.userName}/.bundle/${bundle.name}/${bundle.target}${var.seperator}/${bundle.target}-mlops-stacks-experiment
resources:
# additional job to be deployed in 'test' for cleaning up tests' resources
jobs:
resources_cleanup_job:
name: ${bundle.target}${var.seperator}-mlops-stacks-resources-cleanup-job
max_concurrent_runs: 1
permissions:
- level: CAN_VIEW
group_name: users
tasks:
- task_key: resources_cleanup_job
job_cluster_key: resources_cleanup_cluster
notebook_task:
notebook_path: utils/notebooks/TestResourcesCleanup.py # without ../
base_parameters:
schema_full_name: test.mlops_stacks_demo
seperator: ${var.seperator}
git_source_info: url:${bundle.git.origin_url}; branch:${bundle.git.branch}; commit:${bundle.git.commit}
job_clusters:
- job_cluster_key: resources_cleanup_cluster
new_cluster:
num_workers: 3
spark_version: 13.3.x-cpu-ml-scala2.12
node_type_id: i3.xlarge
custom_tags:
clusterSource: mlops-stack/0.2