Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
a month ago
Databricks Secrets is the valid approach for storing sensitive credentials. Secrets are encrypted, access-controlled, and can be referenced in job parameters and notebooks without exposing values in code or logs. You can follow below
- You can Create a Secret Scope - one time setup.
- Attach & store the various credentials
- Reference Secrets in the Notebooks/Code like below
# SQL Server
sqlserver_url = f"jdbc:sqlserver://{dbutils.secrets.get('scope-name', 'sqlserver-host')};database={dbutils.secrets.get('scope-name', 'sqlserver-database')}"
sqlserver_properties = {
"user": dbutils.secrets.get("scope-name", "sqlserver-user"),
"password": dbutils.secrets.get("scope-name", "sqlserver-password")
}
df_sqlserver = spark.read.jdbc(url=sqlserver_url, table="schema.table", properties=sqlserver_properties)
# Postgres
postgres_url = f"jdbc:postgresql://{dbutils.secrets.get('scope-name', 'postgres-host')}:5432/database"
postgres_properties = {
"user": dbutils.secrets.get("scope-name", "postgres-user"),
"password": dbutils.secrets.get("scope-name", "postgres-password"),
}
df_postgres = spark.read.jdbc(url=postgres_url, table="schema.table", properties=postgres_properties)
# MongoDB
mongo_uri = dbutils.secrets.get("scope-name", "mongodb-connection-string")
df_mongo = spark.read.format("mongodb")\
.option("connection.uri", mongo_uri)\
.option("database", "db_name")\
.option("collection", "collection_name")\
.load()More details here