cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

DLT PipelineID/PipleLineName values from the TASK1 should get passed to TASK2 notebook (Non-DLT)

skumarrm
New Contributor II

DLT PipelineID/PipleLineName values from the TASK1 should get passed to TASK2 notebook (Non-DLT)

TASK1(DLT)---> TASK2(Non-DLT)

How to pass the parameters to TASK2 from TASK1. I need to get the DLT task notebook pipelineID,pipelineName and pass to TASK2 notebook parameters?

Need help currently, the below NOT worked

pipeline_id = dbutils.jobs.taskContext().pipelineId()
pipeline_name = dbutils.jobs.taskContext().pipelineName()

2 REPLIES 2

MuthuLakshmi
Databricks Employee
Databricks Employee

@skumarrm 
Please try the below:

Set Up Task Parameters:

    • In the job configuration, you can set up task parameters to pass values from one task to another.
    • For TASK1 (DLT), ensure it outputs the PipelineID or PipelineName.

Use Task Parameters in TASK2:

    • In the notebook for TASK2, use the dbutils.widgets API to retrieve the parameters passed from TASK1.

Hereโ€™s an example of how you can set this up:

TASK1 (DLT Pipeline)

  • Ensure your DLT pipeline is configured and running.

TASK2 (Non-DLT Notebook)

  • Create a notebook with the following code to retrieve the parameters:

# Retrieve the PipelineID or 

PipelineName passed from TASK1

pipeline_id = dbutils.widgets.get

("PipelineID")

pipeline_name = dbutils.widgets.get

("PipelineName")

# Use the parameters in your notebook 

logic

print(f"PipelineID: {pipeline_id}")

print(f"PipelineName: {pipeline_name}")

# Your notebook logic here

Job Configuration

  • In the job configuration, set up TASK1 to run the DLT pipeline.
  • Add TASK2 and configure it to run the notebook created above.
  • In the task settings for TASK2, add parameters to retrieve the PipelineID and PipelineName from TASK1.

Thank you @MuthuLakshmi for sharing your inputs. Hope I followed it and again got a error -  Failed to resolve references: Task value referenced by 'tasks.DLT.values.pipeline_id' was not found.

I have attached the screenshots of the notebook logic.

Step 1:

DLT_notebook:

In command 3,

                             We are assigning the id to pipeline_id variable and to pass the value in the job task parameter (pipeline_id)

sriram_kumar_0-1731333850989.png

Non_DLT notebook:

      We are getting the value from the DLT notebook passed through task parameter.

sriram_kumar_1-1731333874132.png

Step 2:

In the job configuration setting the parameter to take the value from the DLT notebook.

sriram_kumar_2-1731333890058.png

The job failed with the error below:

sriram_kumar_3-1731333901735.pngsriram_kumar_4-1731333908247.png

When we checked with other people who worked in DLT, they told dbutils.job.taskValues will not work in DLT pipeline.

Could you please help on this and kindly let us know if we I'm missing something.
Thank you in advance.

 

 


 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group