Deny assignment modification to allow attach/detach of disks in azure databricks

[Lucidity]

Our application does storage autoscaling on Azure. We would like to deploy our solution with Azure databricks. But even though the service principal associated with our application has the necessary roles and permissions to attach/detach a disk from a VM, its unable to do because of the deny assignment created by databricks on azure. Is there a way to modify the deny assignment or get the service principal associated with our appplication in the exludePrincipal section of the deny assignment.

If this is not possible, does databricks itself provides api's to attach / detach a disk from a VM

[Ayushi_Suthar · Accepted solution]

Hi @Lucidity , Thanks for bringing up your concerns, always happy to help 😁

According to the provided information, it seems that you are attempting to modify the disk resources. However, access is being denied due to a deny assignment. This denial is occurring because the resources are within a managed resource group (MRG). When attempting to make changes to a managed resource group or its associated resources, users encounter system deny assignment errors.

Under a managed resource group, users have restricted access. These MRGs are established during workspace creation and are under the control of Databricks management. No alterations or modifications can be applied to the MRG itself or the resources within it.

Please refer this document for the more details: https://learn.microsoft.com/en-us/azure/role-based-access-control/deny-assignments

Please let me know if this helps and leave a like if this helps, followups are appreciated.
Kudos
Ayushi

[Lucidity]

Thank you for your reply

Is there any way databricks provides to bypass the deny assignment for specific apps? I noticed in the deny assignment unity-catalog-access-connector has been provided exlusion under the excludePrincipals section. is there a way to get our application also excluded from the deny assignment?

Are you aware of any API's databricks provides to attach / detach disks from a VM part of databricks cluster part of managed resource group?


Apart from this, would be really helpful if you can suggest any other alternatives

thanks 

[Kaniz_Fatma]

Hi @Lucidity, 

Bypassing Deny Assignment for Specific Apps:

• Unfortunately, on the managed resource group created by Azure Databricks, you cannot directly override the denied assignment even if you are the owner. This resource is managed by Databricks, and it restricts direct access to data due to system information stored inside the storage account.
• Attempting to bypass this restriction will result in an error similar to the one you encountered.
• The deny assignment is specifically designed to prevent unauthorized access to certain resources within the managed environment.
• If you need to access specific resources, consider creating new databases with explicit locations pointing to your own storage accounts for production data. This approach allows you to manage access more flexibly.

APIs for Attaching/Detaching Disks from VMs in Databricks cluster:

• Databricks itself does not provide direct APIs for attaching or detaching disks from virtual machines (VMs) within a managed resource group.
• However, you can use Azure Compute REST APIs to achieve this. Specifically, the attachDetachDataDisks operation allows you to attach and detach data disks to/from....
• You can programmatically invoke this API to manage data disks associated with your VMs in the Databricks cluster.

Other Alternatives:

• If you’re looking for alternatives beyond direct disk management, consider the following:
  • Azure Data Lake Storage Gen2: Use separate storage accounts for your data lake storage. Managed tables are created by default on DBFS under dbfs:/user/hive/warehouse/, but for production data, create new databases with explicit locations pointing to your own storage accounts.
  • Custom Roles and Permissions: Explore custom role assignments and permissions within Azure. You can create custom roles that allow specific actions while adhering to security requirements.
  • Resource Group Management: If you no longer need certain resource groups, you can delete them. However, ensure that any deny assignments are addressed before deletion23.

Remember that security and access control are critical aspects, so choose the approach that aligns best with your requirements and organizational policies. If you have further questions or need additional guidance, feel free to ask! 😊