Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-01-2024 03:54 AM
Don't know if this is what you are trying to achieve, but during my limited testing I managed to deploy an extra notebook to a DLT pipeline to **only** stg by referencing it as an additional library:
targets:
dev:
default: true
resources:
pipelines:
sales_pipeline:
development: true
stg:
workspace:
host: https://xxxx.x.azuredatabricks.net/ #Replace with the host address of your stg environment
resources:
pipelines:
sales_pipeline:
libraries:
#Adding a Notebook to the DLT pipeline that tests the data
- notebook:
path: "./50_tests/10_integration/DLT-Pipeline-Test.py"
development: true
prod:
workspace:
host: https://xxx.x.azuredatabricks.net/ #Replace with the host address of your prod environment
resources:
pipelines:
sales_pipeline:
development: false
#Update the cluster settings of the DLT pipeline
clusters:
- autoscale:
min_workers: 1
max_workers: 2The actual asset is deployed to all targets, but the pipeline in where it is referenced and ran is target specific.
More info and source code here if you want to test: ossinova/databricks-asset-bundles-demo: A demo of using databricks asset bundles (github.com)