cancel
Showing results for 
Search instead for 
Did you mean: 
Warehousing & Analytics
Engage in discussions on data warehousing, analytics, and BI solutions within the Databricks Community. Share insights, tips, and best practices for leveraging data for informed decision-making.
cancel
Showing results for 
Search instead for 
Did you mean: 

Access specific input item of For Each Tasks

Hania__b
New Contributor

Hi,

I think I have a similar issue to the one in this post, but the answer isn't detailed enough for me.


I have a list defined in my first task, which contains the items I want to iterate through [1,2,3,4]. When I use it as Inputs to the For Each framing task, I can see it as Input, e.g. Input = 1, Input = 2 but can't access it in the notebook so that I could use it as a variable in the second task (not the entire list, but just the item in the list).

Can you explain your comment, with an example?

I've tried using a Job Parameter, but when it prints in the notebook, it seems to output the same parameter value for every iteration in the looper, but I might be applying it wrongly.

1 ACCEPTED SOLUTION

Accepted Solutions

Walter_C
Databricks Employee
Databricks Employee

To access the individual items from your list [1, 2, 3, 4] in the notebook for each iteration of the For Each task, you can use the {{input}} reference. Here’s a step-by-step example to help you understand how to achieve this:

  1. Define the For Each Task:

    • Create a new For Each task in your job configuration.
    • Set the inputs field to your list: "[1, 2, 3, 4]".
    • Configure the nested task that will run for each item in the list.
  2. Configure the Nested Task:

    • In the nested task, use the {{input}} reference to access the current item in the iteration.
    • For example, if you are using a notebook task, you can pass the {{input}} as a parameter to the notebook.
  3. Access the Parameter in the Notebook:

    • In your notebook, use dbutils.widgets.get to retrieve the parameter value.

Here’s an example configuration:

{
  "name": "Process Items",
  "tasks": [
    {
      "task_key": "process_items",
      "for_each_task": {
        "inputs": "[1, 2, 3, 4]",
        "task": {
          "task_key": "process_item_iteration",
          "notebook_task": {
            "notebook_path": "/path/to/your/notebook",
            "base_parameters": {
              "item": "{{input}}"
            }
          }
        }
      }
    }
  ]
}

View solution in original post

3 REPLIES 3

Walter_C
Databricks Employee
Databricks Employee

To access the individual items from your list [1, 2, 3, 4] in the notebook for each iteration of the For Each task, you can use the {{input}} reference. Here’s a step-by-step example to help you understand how to achieve this:

  1. Define the For Each Task:

    • Create a new For Each task in your job configuration.
    • Set the inputs field to your list: "[1, 2, 3, 4]".
    • Configure the nested task that will run for each item in the list.
  2. Configure the Nested Task:

    • In the nested task, use the {{input}} reference to access the current item in the iteration.
    • For example, if you are using a notebook task, you can pass the {{input}} as a parameter to the notebook.
  3. Access the Parameter in the Notebook:

    • In your notebook, use dbutils.widgets.get to retrieve the parameter value.

Here’s an example configuration:

{
  "name": "Process Items",
  "tasks": [
    {
      "task_key": "process_items",
      "for_each_task": {
        "inputs": "[1, 2, 3, 4]",
        "task": {
          "task_key": "process_item_iteration",
          "notebook_task": {
            "notebook_path": "/path/to/your/notebook",
            "base_parameters": {
              "item": "{{input}}"
            }
          }
        }
      }
    }
  ]
}

MariuszK
Contributor

Hi,
You need a notebook to pass list:

dbutils.jobs.taskValues.set(key = "list", value = your_list)

In foreach you need a notebook with widgest to read passed parameters:

dbutils.widgets.text("param_1", "")
param_1 = dbutils.widgets.get("param_1")

 

 

Hania__b
New Contributor

Thank you both very much, I've nailed it 😁. I have accepted Walter_C's answer as solution because Step 2 is what I was missing. Thanks MariuszK as well for your contribution.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group