10-26-2023 07:06 AM
When generating the standard setup with databricks bundle init we will get databricks.yml that references resources/*. The targets are set in the databricks.yml and the resources (pipelines and jobs) are set in different files.
I have dlt pipelines that I want to run continuously in the production workspace but, to save cost, want to run triggered in the dev workspace. My pipelines in Azure Devops deploy the code first to dev and then to prod using the databricks bundle deploy -t xxx.
Is there a best practice for how to implement the differences?
I tried adding an if statement but it doesn't seem to be working
10-30-2023 12:38 AM
Hi @Mathias ,
To configure the continuous flag for your Delta Live Tables pipelines differently for dev and prod workspaces, it's best to use separate YAML files for each environment. Here's the approach:
Create Separate Pipeline YAML Files: Generate distinct YAML files (e.g., dlt_pipeline_dev.yml
and dlt_pipeline_prod.yml
) for your Delta Live Tables pipelines in each environment. Set the continuous flag accordingly in these files.
dlt_pipeline_dev.yml:
apiVersion: 1.3
name: my_dlt_pipeline_dev
pipelines:
- name: my_pipeline_dev
continuous: false
source_path: /mnt/dev/source/path/
destination_path: /mnt/dev/destination/path/
schema_metaline: "col1 int, col2 string"
dlt_pipeline_prod.yml:apiVersion: 1.3
name: my_dlt_pipeline_prod
pipelines:
- name: my_pipeline_prod
continuous: true
source_path: /mnt/prod/source/path/
destination_path: /mnt/prod/destination/path/
schema_metaline: "col1 int, col2 string"
2. Deploy the Appropriate Pipeline File: When deploying your Delta Live Tables pipelines to each environment, select the relevant pipeline file using the -f
option. For example:
For Dev:
databricks bundle deploy -t dev -f dlt_pipeline_dev.yml
For Prod:
databricks bundle deploy -t prod -f dlt_pipeline_prod.yml
By using separate YAML files for each environment, you can ensure that the continuous
flag is correctly set for each pipeline, tailored to the specific requirements of that environment.
03-01-2024 07:18 AM
@Kaniz_Fatma The YAML snippets you included aren't valid bundle configurations. The `-f` flag also doesn't exist.
DABs are always deployed in their entirety, not selectively.
The configuration syntax reference can be found here: https://docs.databricks.com/en/dev-tools/bundles/settings.html
11-02-2023 07:46 AM
In our case we are using build pipelines in Azure Devops do deploy the solution, which consist or multiple different jobs and dlt pipelines. We are using Databricks bundles as created by databricks bundle init, which creates one databricks.yml and a Resources -folder, where the individual pipeline specifications reside.
To give context, our build pipeline looks like this:
The deploy-steps.yml looks like this:
Any recommendations on how to specify different settings per environment in this case?
03-01-2024 07:22 AM
It is possible to use target overrides to customize resources based on the target you're deploying to.
Documentation can be found here: https://docs.databricks.com/en/dev-tools/bundles/settings.html#targets
An example of this pattern can be found here: https://github.com/databricks/cli/blob/main/bundle/tests/override_pipeline_cluster/databricks.yml
Instead of the `clusters` or `name` field, you would include the `continuous` field.
07-17-2024 08:09 AM
That's exactly my words! I'd not be surprised if this were the author of DAB judging by the nickname (https://github.com/databricks/cli/commits?author=pietern) 😉
02-23-2024 11:19 AM
-f is unknown shorthand flag for databricks bundle deploy. Any workaround on how to deploy different jobs with different targets?
05-27-2024 07:28 AM
I also like to know the solution to this problem
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group