Customer Managed Keys in Databricks (AWS)

[Phani1]

Hi Databricks Team,

Could you please provide me the detailed steps on how to be enabled customer managed keys in databricks (AWS) Account, if there is any video on it that would be great helpful.

Regards,

Phanindra

[Kaniz]

Hi @Phani1, Enabling customer-managed keys (CMKs) in Databricks on AWS involves securing and controlling access to encrypted data. 

Here are the steps to set up CMKs:

Prerequisites:

• Ensure your Databricks workspace is on the E2 version of the Databricks Platform.
• You need to be on the Enterprise pricing tier to access these features.

Types of Data and Locations:

• Databricks has two customer-managed key features:
  • Customer-managed keys for managed services: Used for various data types in the control plane.
  • Customer-managed keys for workspace storage: Used for data stored in your workspace’s S3 bucket.

Data Types and Corresponding Key Features:

• Here’s a breakdown of which key feature to use for different types of data:
  • Notebook source and metadata: Use customer-managed keys for managed services.
  • Personal access tokens (PAT) or other credentials for Git integration: Use customer-managed keys for managed services.
  • Secrets stored by the secret manager APIs: Use customer-managed keys for managed services.
  • Databricks SQL queries and query history: Utilize customer-managed keys for managed services.
  • Remote EBS volumes for Databricks Runtime cluster nodes: Apply customer-managed keys (applies only to classic compute plane resources).
  • Workspace storage:
    • Customer-accessible DBFS root data: Use customer-managed keys for workspace storage.
    • Job results, Databricks SQL query results, MLflow Models, and Delta Live Table: All stored in your workspace’s S3 bucket, so use customer-managed keys for workspace storage.
    • Interactive notebook results: Configure Databricks to store them in your AWS account using the appropriate key feature based on storage location.

Serverless Compute and Customer-Managed Keys:

• For Databricks SQL Serverless, you can use:
  • Customer-managed keys for managed services for SQL queries and history.
  • Customer-managed keys for your workspace’s S3 bucket (including root DBFS storage for SQL results).
  • SQL warehouses do not use customer-managed keys for EBS storage encryption on compute nodes.

For a more detailed walkthrough, refer to the official Databricks documentation on customer-managed ....

🚀Our End-of-Year Community Survey is here! Please take a few moments to complete the survey. Your feedback matters!