- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-22-2026 04:09 AM
I'd think about this as a separation of concerns:
Secrets are for sensitive values (usernames, passwords, tokens, connection URIs).
Job parameters are for runtime values (connection name, database, schema, table, query, collection, source system).
In most cases I would avoid passing credentials as job parameters entirely. Job parameters are designed for runtime configuration and can be overridden at execution time, so they shouldn't be treated as a secure credential store.
My usual recommendation would be:
SQL Server / PostgreSQL
Store credentials in a Databricks Secret Scope.
If you're using Unity Catalog, consider a JDBC Unity Catalog Connection so the URL, driver, and credentials are managed centrally.
Pass only non-sensitive values (connection name, table, schema, query, etc.) as job parameters.
MongoDB
Store the connection URI in a Secret Scope.
Pass database and collection names as job parameters.
Retrieve the secret at runtime with dbutils.secrets.get(...).
This gives you a clean pattern where credentials are managed once and jobs remain fully parameterized without exposing secrets in code, job definitions, or run configurations.
So the architecture becomes:
Secret Scope / UC Connection → credentials
Job Parameters → database, schema, table, query, collection, source system
That separation tends to scale much better as the number of sources and jobs grows.