- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-27-2024 10:55 AM - edited 11-27-2024 11:01 AM
Hi @jeremy98 ,
Unfortunately, you cannot use variables in include mapping. What I was trying to suggest, you should place resources with "common logic" shared across all environments in your yml file that you're passing to include mapping. Then environment specific settings/workflows should be overriden/added in targets mapping of indvidual environments.
For example, have a look at below example 1 from documentation. They defined some job in top-level resources mapping (it could come from include mapping, doesn't matter for sake of an example).
Next, for development environment they added some specific configuration only for dev, basically overriding "core" job logic that is coming from top-level resources. But also in targets mapping you can add completly new job definition that will only be deployed to specific environment.
So, when a target mapping specifies a workspace, artifacts, or resources mapping, and a top-level workspace, artifacts, or resources mapping also exists, then any conflicting settings are overridden by the settings within the target,
PS: The problem you're dealing with is quite common. It would be great if databricks team add ability to use variables in include mapping or allow to override include mapping in targets mapping.
Anyway, you can also take a look at below threads and try different approaches. Maybe you can use sync mapping in a clever way?
- Variables in databricks.yml "include:" - Asset Bun... - Databricks Community - 78893
- Databricks Bundles - How to select which jobs reso... - Databricks Community - 62023