topic Pipelines not included in Databricks Asset Bundles deployment in Data Engineering

Pipelines not included in Databricks Asset Bundles deployment

Charansai — Thu, 13 Nov 2025 16:41:42 GMT

Hi all,

I’m working with Databricks Asset Bundles (DAB) to build and deploy Jobs and pipelines across multiple environments in Azure Databricks.

I can successfully deploy Jobs using bundles.
However, when I try to deploy pipelines, I notice that the bundle does not include pipeline resources in the build/deploy process at all.

From what I understand, pipelines need to be explicitly defined in the bundle configuration under the resources section. Unlike Jobs, they aren’t automatically picked up unless declared. If the pipeline was originally created in the Databricks UI or API, I may need to run databricks bundle generate to produce the YAML definition and then add it to the bundle.

Has anyone else run into this issue? Is there a recommended way to structure pipeline resources in the bundle so they are consistently deployed across environments?

Thanks in advance!

Re: Pipelines not included in Databricks Asset Bundles deployment

cdn_yyz_yul — Thu, 13 Nov 2025 20:10:45 GMT

This example helped me to deploy ETL pipelines as tasks in jobs to different workspaces.

bundle-examples/lakeflow_pipelines_python at main · databricks/bundle-examples · GitHub

Re: Pipelines not included in Databricks Asset Bundles deployment

Coffee77 — Fri, 14 Nov 2025 08:36:14 GMT

As per this documentation https://docs.databricks.com/aws/en/dev-tools/bundles/resources#pipeline you should be able to do it whit latest CLI version. Check you have that latest version.

Here is a sample databricks.yml configuration file -> https://github.com/databricks/bundle-examples/blob/main/lakeflow_pipelines_python/databricks.yml

Here is a sample pipeline definition -> https://github.com/databricks/bundle-examples/blob/main/lakeflow_pipelines_python/resources/pipelines_python_etl.pipeline.yml Notice how "pipelines_python_etl" is set as uniquue key for this pipeline. It means, DAB will overwrite whatever pipeline with that key or create a new one if not exist.

I hope this helps.