In your configuration, youโve defined a default job that uses a tag to determine which version of the notebooks to pull from your remote Git repository. However, for the โdevโ environment, youโd like to use a branch instead of a tag. The challenge is that when you define a branch in the target configuration for โdev,โ it still retains the tag, leading to deployment issues.
To achieve your desired behavior, you can modify your configuration as follows:
-
Default Job Configuration (default_job.yml):
- Keep the existing configuration as it is, using the
git_tag
to pull the notebooks.
- This configuration will be used for all environments except โdev.โ
-
Development Environment Configuration (dev.yml):
- In the
git_source
section for the specific job (letโs call it job_name
), set only the git_branch
without specifying the git_tag
.
- This ensures that when deploying to the โdevโ environment, the tag is ignored, and only the specified branch is used.
Hereโs how your modified configuration would look:
resources:
jobs:
job_name:
name: "name"
...
git_source:
git_url: "<git_url>"
git_provider: "<provider>"
git_tag: "<tag>"
...
targets:
dev:
mode: development
...
resources:
jobs:
job_name:
git_source:
git_branch: "<branch>"
With this setup, when deploying to the โdevโ environment, Databricks will use the specified branch and ignore any associated tags. Your git_source
configuration for the โdevโ environment will look like this:
"git_source": {
"git_branch": "<branch>",
"git_provider": "<provider>",
"git_url": "<git_url>"
}
Remember to replace <branch>
, <provider>
, and <git_url>
with the actual values relevant to your project.
Happy deploying! ๐