Dear Community,
I have a deployment issue after restructuring my project.
Previously, our project was organized with the following structure:
project/src/
project/resources/
project/databricks.yml
As part of an optimization effort, we have transitioned to a monorepo structure with multiple sub-projects:
All resources, including Delta Live Tables (DLT) pipelines that were deployed in the earlier structure, have now been moved to project1/.
Deployment Issue
Following this restructuring, we encountered a problem when deploying with Databricks Asset Bundle (DAB). Specifically:
Error: cannot create pipeline: The pipeline name '[user] bronze__pipeline' is already used by another pipeline. This check can be skipped by setting `allow_duplicate_names = true` in the request.
with databricks_pipeline.bronze_pipeline,
on bundle.tf.json line 315, in resource.databricks_pipeline.bronze__pipeline:
315: },
--> It seems that DAB does not recognize the pipelines as the same resources after the folder restructuring. DAB treats them as new resources rather than updates to existing ones.
Questions:
How can we better understand Databricks Asset Bundle (DAB) behavior when restructuring projects?
What steps can be taken to ensure that we continue using the same resources, namely the same DLT pipelines, after restructuring the project?