Global ini file to reference Databricks-backed secrets (not Azure)

[brian999]

Is there a way to create a global ini file that will reference databricks-backed secrets? Not from Azure, we use databricks on AWS.

[Kaniz]

Hi @brian999, When working with Databricks on AWS, you can create a global initialization script that references Databricks-backed secrets.

Let’s break down the steps:

1. Create a Secret in a Databricks-Backed Scope:

   • To create a secret, you can use the Databricks CLI. Here’s an example command:

     databricks secrets put-secret --json '{
       "scope": "<scope-name>",
       "key": "<key-name>",
       "string_value": "<secret>"
     }'
     

     Replace <scope-name>, <key-name>, and <secret> with appropriate values¹.

2. List Secrets:

   • To view existing secrets within a scope, use the following command:

     databricks secrets list-secrets <scope-name>
     

3. Read a Secret:

   • While you create secrets using the REST API or CLI, you must use the Secrets utility (dbutils.secrets) in a notebook or job to read a secret.

4. Use Secrets in Spark Configuration Properties or Environment Variables:

   • You can reference a secret in a Spark configuration property or environment variable. Retrieved secrets are redacted from notebook output and Spark driver and executor logs.
   • Keep in mind the security implications:
     • If table access control is not enabled on a cluster, any user with appropriate permissions can read Spark configuration properties from within a notebook.
     • Databricks recommends enabling table access control or managing access to secrets using secret scopes.
     • Even when table access control is enabled, users with certain permissions can still read cluster environment variables from within a notebook.
     • Secrets are not redacted from the Spark driver log stdout and stderr streams¹.

If you need further assistance, feel free to ask! 🚀