Schedule job runs with different parameters?

[129876]

Is it possible to schedule different runs for job with parameters? I have a notebook that generates data based on the supplied parameter but would like to schedule runs instead of manually starting them. I assume that this would be possible using the REST API but would like to avoid having my token in plaintext. Would git integration have something for this?

[Debayan]

You can pass parameters for your task. Each task type has different requirements for formatting and passing the parameters.

https://docs.databricks.com/workflows/jobs/jobs.html#create-a-job

REST API can also pass parameters fro jobs. Tokens replace passwords in an authentication flow and should be protected like passwords. To protect tokens, Databricks recommends that you store tokens in:

• Secret management and retrieve tokens in notebooks using the Secrets utility (dbutils.secrets).
• A local key store and use the Python keyring package to retrieve tokens at runtime.

Please refer: https://docs.databricks.com/dev-tools/api/latest/authentication.html

Also, you can store the tokens in .netrc file and use them in curl : https://docs.databricks.com/dev-tools/api/latest/authentication.html#store-tokens-in-a-netrc-file-an...

For GIT integration please refer: https://docs.databricks.com/repos/index.html.

Please let us know if this helps.

[129876]

Yes I am aware of these. I wanted to know if I can schedule tasks with different parameters, is there any functionality in Databricks that would allow this?

[Kaniz]

Hi @k.b.​,

To Run a job with different parameters,

You can use Run Now with Different Parameters to re-run a job with different parameters or values for existing parameters.

• Click the down arrow next to Run Now and select Run Now with Different Parameters or, in the Active Runs table, click Run Now with Different Parameters. Enter the new parameters depending on the type of task.

• Notebook: You can enter parameters as key-value pairs or JSON objects. You can use this dialogue to set the values of widgets.

• JAR and spark-submit: You can enter a list of parameters or a JSON document. The provided parameters are merged with the default parameters for the triggered run. If you delete keys, the default parameters are used. You can also add task parameter variables for the run.

• Click Run.

Source

[129876]

I want to schedule this process daily. To run automatically.

[Kaniz]

Hi @k.b.​, You can create and manage notebook jobs directly in the notebook UI, as given in this article.

You can create and manage schedules if a notebook is already assigned to one or more jobs.

If a notebook is not assigned to a job, you can create a job and a schedule to run the notebook.

Please let us know if this helps.

[129876]

Not really... I have different values for a single parameter within a notebook. Can I schedule under a single job, runs for each different value?