cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

For each task field

jeremy98
Honored Contributor

Hi community,
I was wondering after passing a list of dict through tasks using .taskValue.set() method, how to maintain the same data type through each task?

Because seems, that when I use the for loop and getting by the parameters each element of the dict that I passed I need to cast again the values, is it possible to maintain the data type through each task?

5 REPLIES 5

Alberto_Umana
Databricks Employee
Databricks Employee

jeremy98
Honored Contributor

yes, I reviewed this, but this is a json serialised that could be a problem if a want to maintain the same data type through each task or not?

Alberto_Umana
Databricks Employee
Databricks Employee

Yeah, to ensure that the data types are maintained, you can convert the values to the desired types after deserialization. This is necessary because JSON does not distinguish between integers and floats, and all numbers are deserialized as floats
The dbutils.jobs.taskValues.set() method serializes the data to JSON, and dbutils.jobs.taskValues.get() deserializes it back

Ok, but if I'm using a for each task, and catch each input from the email_info value that I was passing:

          for_each_task:
            inputs: "{{tasks.get_email_infos.values.email_info}}"
            task:
              task_key: process_export_email_iteration

for catching each parameter:

              notebook_task:
                notebook_path: ../notebook/workflow/client_export.py
                base_parameters:
                  function: export_email
                  env: ${bundle.target}
                  id: "{{input.id}}"
                  rendered_next_run: "{{input.rendered_next_run}}"

in the task I need to use the widget and convert again, idk if I explained well my problem that I was having using this path with "for loop"

jeremy98
Honored Contributor

I solved it, passing the variable as a list of string (dict) as input