Pipelines not included in Databricks Asset Bundles deployment

[Charansai]

Hi all,

I’m working with Databricks Asset Bundles (DAB) to build and deploy Jobs and pipelines across multiple environments in Azure Databricks.

• I can successfully deploy Jobs using bundles.

• However, when I try to deploy pipelines, I notice that the bundle does not include pipeline resources in the build/deploy process at all.

From what I understand, pipelines need to be explicitly defined in the bundle configuration under the resources section. Unlike Jobs, they aren’t automatically picked up unless declared. If the pipeline was originally created in the Databricks UI or API, I may need to run databricks bundle generate to produce the YAML definition and then add it to the bundle.

Has anyone else run into this issue? Is there a recommended way to structure pipeline resources in the bundle so they are consistently deployed across environments?

Thanks in advance!

[cdn_yyz_yul]

This example helped me to deploy ETL pipelines as tasks in jobs to different workspaces.

bundle-examples/lakeflow_pipelines_python at main · databricks/bundle-examples · GitHub

[Coffee77]

As per this documentation https://docs.databricks.com/aws/en/dev-tools/bundles/resources#pipeline you should be able to do it whit latest CLI version. Check you have that latest version.

Here is a sample databricks.yml configuration file -> https://github.com/databricks/bundle-examples/blob/main/lakeflow_pipelines_python/databricks.yml

Here is a sample pipeline definition -> https://github.com/databricks/bundle-examples/blob/main/lakeflow_pipelines_python/resources/pipeline...  Notice how "pipelines_python_etl" is set as uniquue key for this pipeline. It means, DAB will overwrite whatever pipeline with that key or create a new one if not exist.

I hope this helps.