How to store credentials in Databricks and assign them to job parameters

[shan-databricks]

I am using SQL Server, Postgres, and MongoDB as data sources, connecting through Spark and JDBC connector. I would like to store the credentials and connection details in Databricks, pass them as job parameters, and need guidance on possible approaches to achieve this.

[balajij8]

Databricks Secrets is the valid approach for storing sensitive credentials. Secrets are encrypted, access-controlled, and can be referenced in job parameters and notebooks without exposing values in code or logs. You can follow below

1. You can Create a Secret Scope - one time setup.
2. Attach & store the various credentials
3. Reference Secrets in the Notebooks/Code like below

# SQL Server
sqlserver_url = f"jdbc:sqlserver://{dbutils.secrets.get('scope-name', 'sqlserver-host')};database={dbutils.secrets.get('scope-name', 'sqlserver-database')}"
sqlserver_properties = {
    "user": dbutils.secrets.get("scope-name", "sqlserver-user"),
    "password": dbutils.secrets.get("scope-name", "sqlserver-password")
}

df_sqlserver = spark.read.jdbc(url=sqlserver_url, table="schema.table", properties=sqlserver_properties)

# Postgres
postgres_url = f"jdbc:postgresql://{dbutils.secrets.get('scope-name', 'postgres-host')}:5432/database"
postgres_properties = {
    "user": dbutils.secrets.get("scope-name", "postgres-user"),
    "password": dbutils.secrets.get("scope-name", "postgres-password"),
}

df_postgres = spark.read.jdbc(url=postgres_url, table="schema.table", properties=postgres_properties)

# MongoDB
mongo_uri = dbutils.secrets.get("scope-name", "mongodb-connection-string")
df_mongo = spark.read.format("mongodb")\
    .option("connection.uri", mongo_uri)\
    .option("database", "db_name")\
    .option("collection", "collection_name")\
    .load()

More details here

[Ashwin_DSA]

Hi @shan-databricks,

You can do this in Databricks, but it's recommended to separate secrets from runtime parameters.

For SQL Server and PostgreSQL, the modern approach is usually a JDBC Unity Catalog connection. A JDBC UC connection stores the JDBC driver, URL path, and credentials in a governed connection object, works across supported compute types, and keeps credentials hidden from the querying user. In that pattern, your job parameters only need to carry non-sensitive values such as the connection name, schema, table, or query.

If the requirement is read-only access, Databricks generally recommends Lakehouse Federation instead of raw JDBC for supported sources such as PostgreSQL and SQL Server.

For MongoDB, I would usually recommend using the MongoDB Spark connector together with Databricks secrets rather than passing credentials as job parameters. Databricks documents MongoDB as a third-party Spark connector use case, and custom third-party Spark JARs are supported on dedicated clusters only, not on Databricks SQL, serverless, or standard access mode clusters.

More generally, use job parameters only for non-secret runtime values such as source_system, connection_name, database, collection, table, or query. Databricks explicitly notes that job parameters can be overridden at run time and that a default parameter value is not a security control.

If you are using a notebook task, a common pattern is:

• Retrieve secrets with dbutils.secrets.get()
• Retrieve job parameters with dbutils.widgets.get()

Databricks also supports referencing secrets in Spark configuration properties or environment variables using {{secrets/scope/key}}, but this comes with additional security considerations and is usually not the first choice unless you specifically need cluster-level or init-script-based injection.

A few example patterns below may help make this more concrete.

For SQL Server/PostgreSQL using a JDBC Unity Catalog connection:

connection_name = dbutils.widgets.get("connection_name")
table_name = dbutils.widgets.get("table_name")

df = (
    spark.read.format("jdbc")
    .option("databricks.connection", connection_name)
    .option("dbtable", table_name)
    .load()
)

This is the pattern Databricks recommends when migrating existing Spark JDBC code...move the URL and credentials into the connection object, and pass only runtime values like the connection name and table/query from the job.

For Notebook parameters + Databricks secrets:

source_system = dbutils.widgets.get("source_system")
database_name = dbutils.widgets.get("database_name")
username = dbutils.secrets.get(scope="ext-sources", key="username")
password = dbutils.secrets.get(scope="ext-sources", key="password")

This works well when you want the job to stay parameter-driven, but still keep credentials in a secret scope instead of the job definition.

For MongoDB using the Spark connector:

mongo_uri = dbutils.secrets.get(scope="ext-sources", key="mongodb-uri")
database_name = dbutils.widgets.get("database_name")
collection_name = dbutils.widgets.get("collection_name")

df = (
    spark.read.format("mongodb")
    .option("connection.uri", mongo_uri)
    .option("database", database_name)
    .option("collection", collection_name)
    .load()
)

For MongoDB, the connector-based approach is the better fit than trying to force everything through JDBC-style job parameters, especially since Databricks documents MongoDB as a third-party Spark connector use case.

For SQL Server using the bundled connector directly

df = (
    spark.read.format("sqlserver")
    .option("host", dbutils.widgets.get("host"))
    .option("user", dbutils.secrets.get(scope="ext-sources", key="sqlserver-user"))
    .option("password", dbutils.secrets.get(scope="ext-sources", key="sqlserver-password"))
    .option("database", dbutils.widgets.get("database_name"))
    .option("query", dbutils.widgets.get("source_query"))
    .load()
)

Databricks also documents this bundled-connector pattern for SQL Server. If you do this, the same principle applies.. as in...keep secrets in secret scopes, and use job parameters only for non-sensitive runtime inputs.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

[Yogasathyandrun]

I'd think about this as a separation of concerns:

• Secrets are for sensitive values (usernames, passwords, tokens, connection URIs).

• Job parameters are for runtime values (connection name, database, schema, table, query, collection, source system).

In most cases I would avoid passing credentials as job parameters entirely. Job parameters are designed for runtime configuration and can be overridden at execution time, so they shouldn't be treated as a secure credential store.

My usual recommendation would be:

SQL Server / PostgreSQL

• Store credentials in a Databricks Secret Scope.

• If you're using Unity Catalog, consider a JDBC Unity Catalog Connection so the URL, driver, and credentials are managed centrally.

• Pass only non-sensitive values (connection name, table, schema, query, etc.) as job parameters.

MongoDB

• Store the connection URI in a Secret Scope.

• Pass database and collection names as job parameters.

• Retrieve the secret at runtime with dbutils.secrets.get(...).

This gives you a clean pattern where credentials are managed once and jobs remain fully parameterized without exposing secrets in code, job definitions, or run configurations.

So the architecture becomes:

Secret Scope / UC Connection → credentials

Job Parameters → database, schema, table, query, collection, source system

That separation tends to scale much better as the number of sources and jobs grows.