topic Re: For each task field in Data Engineering

For each task field

jeremy98 — Thu, 09 Jan 2025 18:17:40 GMT

Hi community,
I was wondering after passing a list of dict through tasks using .taskValue.set() method, how to maintain the same data type through each task?

Because seems, that when I use the for loop and getting by the parameters each element of the dict that I passed I need to cast again the values, is it possible to maintain the data type through each task?

Re: For each task field

Alberto_Umana — Thu, 09 Jan 2025 18:35:33 GMT

Hi @jeremy98,

Have you reviewed this: https://docs.databricks.com/en/jobs/task-values.html

Re: For each task field

jeremy98 — Thu, 09 Jan 2025 18:36:53 GMT

yes, I reviewed this, but this is a json serialised that could be a problem if a want to maintain the same data type through each task or not?

Re: For each task field

Alberto_Umana — Thu, 09 Jan 2025 19:02:35 GMT

Yeah, to ensure that the data types are maintained, you can convert the values to the desired types after deserialization. This is necessary because JSON does not distinguish between integers and floats, and all numbers are deserialized as floats
The dbutils.jobs.taskValues.set() method serializes the data to JSON, and dbutils.jobs.taskValues.get() deserializes it back

Re: For each task field

jeremy98 — Thu, 09 Jan 2025 20:29:17 GMT

Ok, but if I'm using a for each task, and catch each input from the email_info value that I was passing:

for_each_task: inputs: "{{tasks.get_email_infos.values.email_info}}" task: task_key: process_export_email_iteration

for catching each parameter:

notebook_task: notebook_path: ../notebook/workflow/client_export.py base_parameters: function: export_email env: ${bundle.target} id: "{{input.id}}" rendered_next_run: "{{input.rendered_next_run}}"

in the task I need to use the widget and convert again, idk if I explained well my problem that I was having using this path with "for loop"

Re: For each task field

jeremy98 — Fri, 10 Jan 2025 15:08:21 GMT

I solved it, passing the variable as a list of string (dict) as input