Yogasathyandrun
Contributor

I'd think about this as a separation of concerns:

  • Secrets are for sensitive values (usernames, passwords, tokens, connection URIs).

  • Job parameters are for runtime values (connection name, database, schema, table, query, collection, source system).

In most cases I would avoid passing credentials as job parameters entirely. Job parameters are designed for runtime configuration and can be overridden at execution time, so they shouldn't be treated as a secure credential store.

My usual recommendation would be:

SQL Server / PostgreSQL

  • Store credentials in a Databricks Secret Scope.

  • If you're using Unity Catalog, consider a JDBC Unity Catalog Connection so the URL, driver, and credentials are managed centrally.

  • Pass only non-sensitive values (connection name, table, schema, query, etc.) as job parameters.

MongoDB

  • Store the connection URI in a Secret Scope.

  • Pass database and collection names as job parameters.

  • Retrieve the secret at runtime with dbutils.secrets.get(...).

This gives you a clean pattern where credentials are managed once and jobs remain fully parameterized without exposing secrets in code, job definitions, or run configurations.

So the architecture becomes:

Secret Scope / UC Connection → credentials

Job Parameters → database, schema, table, query, collection, source system

That separation tends to scale much better as the number of sources and jobs grows.

Data Engineer | Apache Spark | Delta Lake | Databricks