topic DLT PipelineID/PipleLineName values from the TASK1 should get passed to TASK2 notebook (Non-DLT) in Data Engineering

DLT PipelineID/PipleLineName values from the TASK1 should get passed to TASK2 notebook (Non-DLT)

skumarrm — Mon, 11 Nov 2024 11:24:37 GMT

DLT PipelineID/PipleLineName values from the TASK1 should get passed to TASK2 notebook (Non-DLT)

TASK1(DLT)---> TASK2(Non-DLT)

How to pass the parameters to TASK2 from TASK1. I need to get the DLT task notebook pipelineID,pipelineName and pass to TASK2 notebook parameters?

Need help currently, the below NOT worked

pipeline_id = dbutils.jobs.taskContext().pipelineId()
pipeline_name = dbutils.jobs.taskContext().pipelineName()

Re: DLT PipelineID/PipleLineName values from the TASK1 should get passed to TASK2 notebook (Non-DLT)

MuthuLakshmi — Mon, 11 Nov 2024 12:54:01 GMT

@skumarrm
Please try the below:

Set Up Task Parameters:

In the job configuration, you can set up task parameters to pass values from one task to another.
For TASK1 (DLT), ensure it outputs the PipelineID or PipelineName.

Use Task Parameters in TASK2:

In the notebook for TASK2, use the dbutils.widgets API to retrieve the parameters passed from TASK1.

Here’s an example of how you can set this up:

TASK1 (DLT Pipeline)

Ensure your DLT pipeline is configured and running.

TASK2 (Non-DLT Notebook)

Create a notebook with the following code to retrieve the parameters:

# Retrieve the PipelineID or

PipelineName passed from TASK1

pipeline_id = dbutils.widgets.get

("PipelineID")

pipeline_name = dbutils.widgets.get

("PipelineName")

# Use the parameters in your notebook

logic

print(f"PipelineID: {pipeline_id}")

print(f"PipelineName: {pipeline_name}")

# Your notebook logic here

Job Configuration

In the job configuration, set up TASK1 to run the DLT pipeline.
Add TASK2 and configure it to run the notebook created above.
In the task settings for TASK2, add parameters to retrieve the PipelineID and PipelineName from TASK1.

Re: DLT PipelineID/PipleLineName values from the TASK1 should get passed to TASK2 notebook (Non-DLT)

sriram_kumar — Mon, 11 Nov 2024 14:21:07 GMT

Thank you @MuthuLakshmi for sharing your inputs. Hope I followed it and again got a error - Failed to resolve references: Task value referenced by 'tasks.DLT.values.pipeline_id' was not found.

I have attached the screenshots of the notebook logic.

Step 1:

DLT_notebook:

In command 3,

We are assigning the id to pipeline_id variable and to pass the value in the job task parameter (pipeline_id)

Non_DLT notebook:

We are getting the value from the DLT notebook passed through task parameter.

Step 2:

In the job configuration setting the parameter to take the value from the DLT notebook.

The job failed with the error below:

When we checked with other people who worked in DLT, they told dbutils.job.taskValues will not work in DLT pipeline.

Could you please help on this and kindly let us know if we I'm missing something.
Thank you in advance.