Query: Extracting Resolved 'Input' Parameter from a Databricks Workflow Run

[Nexusss7]

Hi Everyone,

I have a query regarding extracting the resolved value of the 'Input' parameter (highlighted in yellow in the attached images) from a Databricks workflow run.

The images show:

1. The foreach task receives its input from the Metadata_Fetcher task as:
• Inputs = {{tasks.Metadata_Fetcher.values.table_config_file_path_list}}
2. In the workflow, a foreach task runs child tasks concurrently, and the config_file parameter is dynamically assigned as {{input}}, coming from the foreach task.
3. The workflow run output shows how {{input}} gets resolved for each iteration.

I want to extract this resolved parameter value of ‘input’ for each run instance using Databricks REST API calls or system tables.

Does anyone know how to achieve this? Any insights or suggestions would be greatly appreciated!

[koji_kawamura · Accepted solution]

Hi @Nexusss7 

Out of curiosity, I tried to retrieve the resolved task parameter values. Finding a way to retrieve executed sub-tasks by the for_each task using APIs was challenging. So, I devised a solution using API and system tables.

I simplified the job as follows, focusing on the key parts. The resolved values were successfully retrieved. The whole notebook code is available here. I hope this helps! https://gist.github.com/koji-kawamura-db/ac46d736d411a13cfe70818afdd4b6ec

[Nexusss7]

Hi @koji_kawamura ,
Thanks for the solution! It worked perfectly and was extremely helpful.