topic Re: create databricks scope by reading AWS secrets manager in Data Engineering

create databricks scope by reading AWS secrets manager

thiagoawstest — Mon, 10 Jun 2024 14:05:58 GMT

Hi, I have datbricks on AWS, I created some secrets in AWS Secrets Manger, I would need to create the scopes based on AWS secrets manager.

When I use Azure's Key Vault, when creating the scope, it uses the option -scope-backend-type AZURE_KEYVAULT, but I didn't find it for AWS.

How would I create a scope with which to read the secrets from AWS Secrets Manager?, or would it only be possible via Python code?

Thanks.

Re: create databricks scope by reading AWS secrets manager

Yeshwanth — Mon, 10 Jun 2024 14:15:09 GMT

Hi @thiagoawstest

Step 1: Create Secret Scope
You can create a secret scope using the Databricks REST API as shown below:

python import requests import json # Define the endpoint and headers url = "https://<databricks-instance>/api/2.0/secrets/scopes/create" headers = { "Authorization": "Bearer <your-databricks-token>", "Content-Type": "application/json" } # Define the payload payload = { "scope": "aws-secrets-scope", "initial_manage_principal": "users" } # Make the request response = requests.post(url, headers=headers, data=json.dumps(payload)) if response.status_code == 200: print("Secret scope created successfully.") else: print(f"Failed to create secret scope: {response.text}")

Is this what you are looking for? Please test it once before deploying it in the workload.

Re: create databricks scope by reading AWS secrets manager

thiagoawstest — Mon, 10 Jun 2024 14:53:43 GMT

That wouldn't be it, I created secrets within the AWS secret manager, when I use Azure Key Vault when creating the scope in Databricks, I pass a parameter that reads the key vault, but for AWS I didn't find it to read the AWS secrets manager .

I wanted to understand if it is not supported, or would it only be via boto3?

Re: create databricks scope by reading AWS secrets manager

ceceliac — Mon, 04 Aug 2025 20:43:45 GMT

Hi, did you ever get an answer to this? I am not finding anything in the documentation still and your post was a year ago. We have the same need with an app that runs on AWS and our Databricks instances on AWS, developers want to manage IAM secrets with the AWS secret manager instead of manually updating the secret scope in Databricks.