topic Schedule job runs with different parameters? in Data Engineering

Schedule job runs with different parameters?

129876 — Thu, 03 Nov 2022 14:33:10 GMT

Is it possible to schedule different runs for job with parameters? I have a notebook that generates data based on the supplied parameter but would like to schedule runs instead of manually starting them. I assume that this would be possible using the REST API but would like to avoid having my token in plaintext. Would git integration have something for this?

Re: Schedule job runs with different parameters?

129876 — Fri, 04 Nov 2022 13:46:57 GMT

Yes I am aware of these. I wanted to know if I can schedule tasks with different parameters, is there any functionality in Databricks that would allow this?

Re: Schedule job runs with different parameters?

129876 — Fri, 04 Nov 2022 17:28:07 GMT

I want to schedule this process daily. To run automatically.

Re: Schedule job runs with different parameters?

129876 — Fri, 04 Nov 2022 22:47:42 GMT

Not really... I have different values for a single parameter within a notebook. Can I schedule under a single job, runs for each different value?

Re: Schedule job runs with different parameters?

Debayan — Fri, 04 Nov 2022 07:51:42 GMT

You can pass parameters for your task. Each task type has different requirements for formatting and passing the parameters.

https://docs.databricks.com/workflows/jobs/jobs.html#create-a-job

REST API can also pass parameters fro jobs. Tokens replace passwords in an authentication flow and should be protected like passwords. To protect tokens, Databricks recommends that you store tokens in:

Secret management and retrieve tokens in notebooks using the Secrets utility (dbutils.secrets).
A local key store and use the Python keyring package to retrieve tokens at runtime.

Please refer: https://docs.databricks.com/dev-tools/api/latest/authentication.html

Also, you can store the tokens in .netrc file and use them in curl : https://docs.databricks.com/dev-tools/api/latest/authentication.html#store-tokens-in-a-netrc-file-and-use-them-in-curl

For GIT integration please refer: https://docs.databricks.com/repos/index.html.

Please let us know if this helps.