Need Guidance on Key Rotation Process for Storage Customer-Managed Keys in Databricks Workspace

[Gopi9]

Problem Statement: We are currently utilizing customer-managed keys for Databricks compute encryption at the workspace level. As part of our key rotation strategy, we find ourselves needing to bring down the entire compute/clusters to update storage encryption keys. However, we encounter errors when attempting to update storage encryption keys without shutting down the compute.

Our workspace is shared by multiple application teams, each with automated jobs triggering compute/clusters to start. The process of stopping all workflows/jobs manually is time-consuming. Is there a way to temporarily pass access at workspace level and allow only Databricks admins to facilitate this key rotation process 

Any guidance or best practices on handling key rotations in a shared workspace environment would be greatly appreciated.

Thank you.

[feiyun0112]

Maybe you can use azure key vault to store customer-managed keys

https://learn.microsoft.com/en-us/azure/databricks/security/secrets/secret-scopes#--create-an-azure-...

[Gopi9]

@feiyun0112  Thanks for the reply. the question is how do I stop access temporarily to Databricks workspace for all users except Databricks ADMIN AD group? our workspaces sync with Azure EntraID via SCIM.