Hi all,

We are running into an issue with Databricks Asset Bundles (DAB) when trying to destroy a DLT pipeline. Setup is as follows:

• Two separate service principals:

  • Deployment SP: used by Azure DevOps for deploying bundles.

  • Run_as SP: used for running the DLTs.

• Since CLI v0.267.0, run_as is supported for DLTs in DABs.

• Deployment works fine: DLT pipelines are created as expected via bundle configuration.

• Environment: Databricks CLI v0.267.0

When trying to destroy the bundle, we get:

Error: cannot delete permissions: PERMISSION_DENIED: PERMISSION_DENIED: Only metastore admins can change pipeline owner

• The Deployment SP should be the creator of the pipeline and therefore have manage permissions.

• However, when running databricks pipelines get, the pipeline shows the Run_as SP as the creator — which is not correct.

• Interestingly, pipeline deletion does succeed if we call databricks pipelines delete directly with the Deployment SP. The issue only appears when using databricks bundle destroy. The DAB destroy work as expected, when all DLT's defined in the bundle are deleted via databricks pipelines delete.

Currently, the only workaround we’ve found is to bypass DAB destroy and explicitly call databricks pipelines delete in the CI/CD pipeline. This is not desired by any means, since it makes pipeline management more difficult and risky compared to keeping everything inside DAB lifecycle management.

Has anyone else experienced similar behavior? Why would DAB destroy treat the run_as SP as the creator/owner under the hood, and is there a way to enforce that the Deployment SP is recognized as the pipeline owner so that destroy works consistently?

Thanks in advance!