balajij8
Contributor III

Databricks Secrets is the valid approach for storing sensitive credentials. Secrets are encrypted, access-controlled, and can be referenced in job parameters and notebooks without exposing values in code or logs. You can follow below

  1. You can Create a Secret Scope - one time setup.
  2. Attach & store the various credentials
  3. Reference Secrets in the Notebooks/Code like below
# SQL Server
sqlserver_url = f"jdbc:sqlserver://{dbutils.secrets.get('scope-name', 'sqlserver-host')};database={dbutils.secrets.get('scope-name', 'sqlserver-database')}"
sqlserver_properties = {
    "user": dbutils.secrets.get("scope-name", "sqlserver-user"),
    "password": dbutils.secrets.get("scope-name", "sqlserver-password")
}

df_sqlserver = spark.read.jdbc(url=sqlserver_url, table="schema.table", properties=sqlserver_properties)

# Postgres
postgres_url = f"jdbc:postgresql://{dbutils.secrets.get('scope-name', 'postgres-host')}:5432/database"
postgres_properties = {
    "user": dbutils.secrets.get("scope-name", "postgres-user"),
    "password": dbutils.secrets.get("scope-name", "postgres-password"),
}

df_postgres = spark.read.jdbc(url=postgres_url, table="schema.table", properties=postgres_properties)

# MongoDB
mongo_uri = dbutils.secrets.get("scope-name", "mongodb-connection-string")
df_mongo = spark.read.format("mongodb")\
    .option("connection.uri", mongo_uri)\
    .option("database", "db_name")\
    .option("collection", "collection_name")\
    .load()

More details here