DAB + DLT destroy fails due to ownership/permissions mismatch

[mikvaar]

Hi all,

We are running into an issue with Databricks Asset Bundles (DAB) when trying to destroy a DLT pipeline. Setup is as follows:

• Two separate service principals:

  • Deployment SP: used by Azure DevOps for deploying bundles.

  • Run_as SP: used for running the DLTs.

• Since CLI v0.267.0, run_as is supported for DLTs in DABs.

• Deployment works fine: DLT pipelines are created as expected via bundle configuration.

• Environment: Databricks CLI v0.267.0

When trying to destroy the bundle, we get:

Error: cannot delete permissions: PERMISSION_DENIED: PERMISSION_DENIED: Only metastore admins can change pipeline owner

Observations:

• The Deployment SP should be the creator of the pipeline and therefore have manage permissions.

• However, when running databricks pipelines get, the pipeline shows the Run_as SP as the creator — which is not correct.

• Interestingly, pipeline deletion does succeed if we call databricks pipelines delete directly with the Deployment SP. The issue only appears when using databricks bundle destroy. The DAB destroy work as expected, when all DLT's defined in the bundle are deleted via databricks pipelines delete.

Currently, the only workaround we’ve found is to bypass DAB destroy and explicitly call databricks pipelines delete in the CI/CD pipeline. This is not desired by any means, since it makes pipeline management more difficult and risky compared to keeping everything inside DAB lifecycle management.

Has anyone else experienced similar behavior? Why would DAB destroy treat the run_as SP as the creator/owner under the hood, and is there a way to enforce that the Deployment SP is recognized as the pipeline owner so that destroy works consistently?

Thanks in advance!

[szymon_dybczak]

Hi @mikvaar ,

I know this is really confusing but this is expected behaviour. It's well described in documentation. So,as they stated below, run as can be even use in situations where  original user who created the pipeline has been deactivated - for example, if they left the company. 
So, basically SP configured with run as becomes new owner.

Configure Lakeflow Declarative Pipelines - Azure Databricks | Microsoft Learn

But maybe databricks should think about changing name of that feature. In my opinion it's doing much more than a name suggest and it's confusing, so I'm not surprised that you asked this question.

[sivil]

"But maybe databricks should think about changing name of that feature."

I have to respectfully disagree about renaming the feature. The issue isn't the name itself, but rather that the behavior differs from regular jobs. It would make much more sense for ownership management to be consistent across DLT pipelines and jobs, so users don't have to learn two sets of rules for what should be the same functionality.

I'd encourage Databricks to align these experiences by making the feature behave consistently, and to prioritize feedback from enterprise customers who depend on predictable, unified tools across the platform. This would go a long way toward improving the overall quality and usability of the product.

[mikvaar]

Hi @szymon_dybczak 

Thank you for you response. Coming back to this issue, how did Databricks design DAB's to be used with DLT's? If I have two service principals as stated in the original message, one for deployment and one for running the bundle resources. Bundle deployment works fine, but if the bundle needs to be destroyed, it is not possible via databricks bundle destroy. For me this seems like an bigger problem, if all features of DAB's are not accessible if using DLT + run_as option together. 

[saurabh18cs]

Hi @mikvaar Can you please check if deployment SP have the Can Manage permission (or equivalent) on the DLT pipeline or the workspace. if not can you please explicitly grant access and test? this is a typical scenario when owner sp is different from run_as sp for DLT's

[mikvaar]

Hi @saurabh18cs. The deployment/destroy SP has permission "IS_OWNER" of the pipelines when looking up with databricks pipelines get-permissions. 

As stated above, pipeline deletion succeeds with deployment SP using databricks pipelines delete.

[saurabh18cs]

Hi @mikvaar yes and this highlights a limitation in DAB's destroy logic. Thus asking you to give a try adding one more permission explicitly ( I know owner supersedes this permission but just to give a try and see the impact) . otherwise Databricks can only help here. Thanks

[mikvaar]

Hi @saurabh18cs. I tried explicitly setting CAN_MANAGE permissions for the pipelines to see if destroy works, and the result is the same as with IS_OWNER permissions. As you said, this seems like an issue that only Databricks can resolve.