topic Databricks workflow job in Data Engineering

Databricks workflow job

ctiwari7 — Tue, 26 Nov 2024 15:43:07 GMT

Hi team,

I am trying to execute a workflow job which takes in a parameter as unique identifier. I am using this job parameter to push down to tasks. I was hoping if there is any way for me to use python uuid4() function to generate unique id every time the workflow job runs.

Is there any way to generate uuid on the run time as job parameter.?

Re: Databricks workflow job

Walter_C — Tue, 26 Nov 2024 16:21:15 GMT

The intention is that the same job generates this id and once it is generated that this id is passed as a parameter for other tasks?

Re: Databricks workflow job

Stefan-Koch — Tue, 26 Nov 2024 16:34:41 GMT

hi ctiwari7

A possible way to do that, you create a python file which generates the uuid and then pass it to jobs.taskValues. This is described here: https://docs.databricks.com/en/jobs/task-values.html

As test, I created a python file, with the following content:

import uuid uuid = str(uuid.uuid4()) print(uuid) dbutils.jobs.taskValues.set(key = "uuid", value = uuid)

The file will be added as task in a databricks job.

For the demo, I have a simple notebook which takes a widget as parameter:

In the job, I add a second task, where I refer to the notebook:

In the parameter field: put the following information:
name: uuid
value: {{tasks.create_uuid.values.uuid}}