topic Re: Need Guidance on Key Rotation Process for Storage Customer-Managed Keys in Databricks Workspace in Get Started Discussions

Need Guidance on Key Rotation Process for Storage Customer-Managed Keys in Databricks Workspace

Gopi9 — Thu, 28 Mar 2024 02:19:40 GMT

Problem Statement: We are currently utilizing customer-managed keys for Databricks compute encryption at the workspace level. As part of our key rotation strategy, we find ourselves needing to bring down the entire compute/clusters to update storage encryption keys. However, we encounter errors when attempting to update storage encryption keys without shutting down the compute.

Our workspace is shared by multiple application teams, each with automated jobs triggering compute/clusters to start. The process of stopping all workflows/jobs manually is time-consuming. Is there a way to temporarily pass access at workspace level and allow only Databricks admins to facilitate this key rotation process

Any guidance or best practices on handling key rotations in a shared workspace environment would be greatly appreciated.

Thank you.

Re: Need Guidance on Key Rotation Process for Storage Customer-Managed Keys in Databricks Workspace

feiyun0112 — Thu, 28 Mar 2024 02:57:49 GMT

Maybe you can use azure key vault to store customer-managed keys

https://learn.microsoft.com/en-us/azure/databricks/security/secrets/secret-scopes#--create-an-azure-key-vault-backed-secret-scope

Re: Need Guidance on Key Rotation Process for Storage Customer-Managed Keys in Databricks Workspace

Gopi9 — Thu, 28 Mar 2024 03:02:12 GMT

@feiyun0112 Thanks for the reply. the question is how do I stop access temporarily to Databricks workspace for all users except Databricks ADMIN AD group? our workspaces sync with Azure EntraID via SCIM.