cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

create databricks scope by reading AWS secrets manager

thiagoawstest
Contributor

Hi, I have datbricks on AWS, I created some secrets in AWS Secrets Manger, I would need to create the scopes based on AWS secrets manager.

When I use Azure's Key Vault, when creating the scope, it uses the option -scope-backend-type AZURE_KEYVAULT, but I didn't find it for AWS.

How would I create a scope with which to read the secrets from AWS Secrets Manager?, or would it only be possible via Python code?

Thanks.

2 REPLIES 2

Yeshwanth
Honored Contributor

Hi @thiagoawstest 

Step 1: Create Secret Scope
You can create a secret scope using the Databricks REST API as shown below:

python
import requests
import json

# Define the endpoint and headers
url = "https://<databricks-instance>/api/2.0/secrets/scopes/create"
headers = {
    "Authorization": "Bearer <your-databricks-token>",
    "Content-Type": "application/json"
}

# Define the payload
payload = {
    "scope": "aws-secrets-scope",
    "initial_manage_principal": "users"
}

# Make the request
response = requests.post(url, headers=headers, data=json.dumps(payload))

if response.status_code == 200:
    print("Secret scope created successfully.")
else:
    print(f"Failed to create secret scope: {response.text}")

Is this what you are looking for? Please test it once before deploying it in the workload.

That wouldn't be it, I created secrets within the AWS secret manager, when I use Azure Key Vault when creating the scope in Databricks, I pass a parameter that reads the key vault, but for AWS I didn't find it to read the AWS secrets manager .

I wanted to understand if it is not supported, or would it only be via boto3?

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group