How to use Service Credentials in Unity Catalog for Azure Key Vault RBAC Integration

Azure Databricks now supports a service credential object in Unity Catalog, enabling secure and governed access to external cloud services like Azure Key Vault. A service credential encapsulates a long-term cloud credential, allowing Databricks to access these services while adhering to enterprise-grade governance and security standards. This blog focuses on how to configure service credentials in Unity Catalog to enable Role-Based Access Control (RBAC) for Azure Key Vault integration.

Why Use Service Credentials in Unity Catalog?

Service credentials offer several advantages when integrating Azure Databricks with Azure Key Vault:

• Governance: Centralized management of access permissions through Unity Catalog.
• Security: Long-term credentials are securely managed, reducing the risk of exposure.
• RBAC Compatibility: Fully supports Azure Key Vault's RBAC model, eliminating the need for legacy access policies.
• Granular Permissions: Enables secret-level access control.

Steps to Configure Service Credentials for Azure Key Vault

1. Prerequisites

Before you begin, ensure the following:

• Your Databricks workspace is Unity Catalog-enabled.
• You have Owner or User Access Administrator permissions for the Azure Key Vault.
• The Azure Databricks Access Connector is deployed in your Azure subscription.

2. Set Up the Azure Databricks Access Connector

The Access Connector acts as a managed identity that facilitates secure communication between Databricks and Azure Key Vault.

1. Deploy the Access Connector using the Azure CLI:

   az databricks access-connector create --name "databricks-access-connector-<CUSTOM DESCRIPTION>" \
     --resource-group "my-resource-group" \
     --location "eastus"

         Note: It is recommended to add a custom description for future reference.

      2. Assign the Key Vault Secrets User role to the Access Connector:

• Navigate to your Key Vault in the Azure Portal.
• Go to the "Access Control (IAM)" section.
• Click "Add role assignment."
• Select the Key Vault Secrets User role and assign it to the Access Connector.

3. Create a Service Credential in Unity Catalog

A service credential in Unity Catalog securely encapsulates the long-term credential required to access external services like Azure Key Vault.

1. Open your Databricks workspace and navigate to Unity Catalog.
2. Execute the following SQL command to create a service credential:

   CREATE SERVICE CREDENTIAL ag_kv_service_credential
   TYPE AZURE_KEY_VAULT
   OPTIONS (
     vault_url 'https://<your-key-vault-name>.vault.azure.net/'
   );

This command links your Databricks workspace with your Azure Key Vault using the specified service credential.

4. Access Secrets Programmatically

Once the service credential is configured, you can programmatically retrieve secrets from Azure Key Vault using Databricks notebooks: 

from azure.keyvault.secrets import SecretClient
# Get the service credential provider
credential = dbutils.credentials.getServiceCredentialsProvider('ag_kv_service_credential')
# Initialize the SecretClient
secret_client = SecretClient(
    vault_url="https://<your-key-vault-name>.vault.azure.net/", 
    credential=credential
)
# Retrieve a secret
secret = secret_client.get_secret("<your-seceret-name-in-keyvalut>")
# Print secret details
print(secret.name)
print(secret.value)

This approach ensures that secrets are accessed securely without exposing sensitive credentials in your codebase.

Benefits of Using Service Credentials

1. Enhanced Security: By encapsulating long-term credentials, service credentials minimize risks associated with manual key management.
2. Simplified Governance: Permissions are managed centrally through Unity Catalog, ensuring compliance with organizational policies.
3. Seamless Integration: Service credentials work natively with Azure Key Vault's RBAC model, eliminating dependency on legacy access policies.
4. Scalability: Supports large-scale deployments by enabling secret-level access control across multiple teams and projects.

Troubleshooting Tips

Conclusion

The introduction of service credentials in Unity Catalog revolutionizes how Azure Databricks integrates with external cloud services like Azure Key Vault. By leveraging this feature, organizations can securely govern access to secrets while fully embracing RBAC for enhanced security and compliance. This approach simplifies secret management, reduces operational overhead, and aligns with modern cloud governance practices.