Databricks asset bundle deploys DLT pipelines as duplicate resources

[HoussemBL]

Dear Community,

I have a deployment issue after restructuring my project.

Previously, our project was organized with the following structure:

• project/src/

• project/resources/

• project/databricks.yml

As part of an optimization effort, we have transitioned to a monorepo structure with multiple sub-projects:

• project1/

• project2/

All resources, including Delta Live Tables (DLT) pipelines that were deployed in the earlier structure, have now been moved to project1/.

Deployment Issue

Following this restructuring, we encountered a problem when deploying with Databricks Asset Bundle (DAB). Specifically:

Error: cannot create pipeline: The pipeline name '[user] bronze__pipeline' is already used by another pipeline. This check can be skipped by setting `allow_duplicate_names = true` in the request.

  with databricks_pipeline.bronze_pipeline,
  on bundle.tf.json line 315, in resource.databricks_pipeline.bronze__pipeline:
 315:       },

--> It seems that DAB does not recognize the pipelines as the same resources after the folder restructuring. DAB treats them as new resources rather than updates to existing ones. 

Questions:

1. How can we better understand Databricks Asset Bundle (DAB) behavior when restructuring projects?

2. What steps can be taken to ensure that we continue using the same resources, namely the same DLT pipelines, after restructuring the project?

[ashraf1395]

Hi there, @HoussemBL , 
Though I dont know the exact solution to your answer. To answer your first question - You can look into your .databricks/ folder for better understanding of your dabs behaviour.

- Are you separating your databricks.yml files after news projects are being created or still using the single databicks.yml

[HoussemBL]

Hi @ashraf1395 ,

I am creating two separate databricks.yml for each sub-project.