Hydra configuration and job parameters of DABs

jeremy98 — Wed, 02 Apr 2025 17:50:48 GMT

Hello Community,

I'm trying to create a job pipeline in Databricks that runs a spark_python_task, which executes a Python script configured with Hydra. The script's configuration file defines parameters, such as id.

How can I pass this parameter at the job level in Databricks so that the task picks it up and overrides it using Hydra? And how to use the dbutils.secrets.get through this type of spark_python_task, to retrieve the keys I need?

Re: Hydra configuration and job parameters of DABs

mark_ott — Fri, 07 Nov 2025 16:30:39 GMT

You can pass and override configuration parameters for Hydra in a Databricks spark_python_task by specifying job-level parameters (as arguments) and using environment variables or Hydra’s command line overrides. For accessing secrets with dbutils.secrets.get, ensure your Python script is written to call this Databricks utility directly. Here’s how to achieve both:

Passing Hydra Parameters in Databricks Job

To override Hydra config values like id from a Databricks job:

Use the parameters field in the spark_python_task specification.
Hydra interprets CLI arguments (e.g., id=foobar) and environment variables as overrides.

Example Databricks job JSON:

json

{
  "tasks": [{
    "task_key": "my_hydra_task",
    "spark_python_task": {
      "python_file": "dbfs:/path/to/main.py",
      "parameters": ["id=1234"]
    }
  }]
}

This sends id=1234 to your script; Hydra will override the value if your script is structured to accept it from the CLI (main.py should invoke Hydra using @Hydra.main, which handles CLI overrides automatically).

Using dbutils.secrets.get in Your Script

Inside your Python script:

python

from pyspark.dbutils import DBUtils
# In Databricks, 'dbutils' is automatically available in notebook, for jobs:
dbutils = DBUtils(spark)

secret_value = dbutils.secrets.get(scope="my_scope", key="my_key")

Access your secret as shown, and use it wherever needed.

Example main.py Structure