Databricks asset bundle deploys DLT pipelines as duplicate resources

HoussemBL — Mon, 10 Mar 2025 14:29:15 GMT

Dear Community,

I have a deployment issue after restructuring my project.

Previously, our project was organized with the following structure:

project/src/
project/resources/
project/databricks.yml

As part of an optimization effort, we have transitioned to a monorepo structure with multiple sub-projects:

project1/
project2/

All resources, including Delta Live Tables (DLT) pipelines that were deployed in the earlier structure, have now been moved to project1/.

Deployment Issue

Following this restructuring, we encountered a problem when deploying with Databricks Asset Bundle (DAB). Specifically:

Error: cannot create pipeline: The pipeline name '[user] bronze__pipeline' is already used by another pipeline. This check can be skipped by setting `allow_duplicate_names = true` in the request. with databricks_pipeline.bronze_pipeline, on bundle.tf.json line 315, in resource.databricks_pipeline.bronze__pipeline: 315: },

--> It seems that DAB does not recognize the pipelines as the same resources after the folder restructuring. DAB treats them as new resources rather than updates to existing ones.

Questions:

How can we better understand Databricks Asset Bundle (DAB) behavior when restructuring projects?
What steps can be taken to ensure that we continue using the same resources, namely the same DLT pipelines, after restructuring the project?

Re: Databricks asset bundle deploys DLT pipelines as duplicate resources

ashraf1395 — Tue, 11 Mar 2025 04:23:44 GMT

Hi there, @HoussemBL ,
Though I dont know the exact solution to your answer. To answer your first question - You can look into your .databricks/ folder for better understanding of your dabs behaviour.

- Are you separating your databricks.yml files after news projects are being created or still using the single databicks.yml

Re: Databricks asset bundle deploys DLT pipelines as duplicate resources

HoussemBL — Tue, 11 Mar 2025 09:59:55 GMT

Hi @ashraf1395 ,

I am creating two separate databricks.yml for each sub-project.

topic Re: Databricks asset bundle deploys DLT pipelines as duplicate resources in Data Engineering

Databricks asset bundle deploys DLT pipelines as duplicate resources

Deployment Issue

Questions:

Re: Databricks asset bundle deploys DLT pipelines as duplicate resources

Re: Databricks asset bundle deploys DLT pipelines as duplicate resources