How to capture dlt pipeline id / name using dynamic value reference

ashraf1395
Honored Contributor

‎02-10-2025 11:03 PM

Hi there,

I have a usecase where I want to set the dlt pipeline id in the configuration parameters of that dlt pipeline.

The way we can use workspace ids or task id in notebook task task_id = {{task.id}}/ {{task.name}} and can save them as parameters and can call later as dbutils.widgets.get("task_id").

 

Can we do something similar in dlt pipeline.

like dlt_pipeline_id = {{dlt_pipeline.name}} in the configurations and then use it spark.conf.get("dlt_pipeline_id").

or is there any other way to achieve this

ashraf1395
Honored Contributor

‎02-12-2025 10:06 PM - edited ‎02-12-2025 10:08 PM

Any ideas on this @VZLA, @Alberto_Umana 

0 Kudos

mourakshit
Databricks Employee
Databricks Employee

‎02-13-2025 11:02 AM

 

The answer is yes, you can achieve this in a DLT (Delta Live Tables) pipeline. Here are a few ways to do it:
Method 1: Using {{dlt_pipeline.name}} in the configuration
You can use the {{dlt_pipeline.name}} syntax in your DLT pipeline configuration, just like you would in a notebook task. This will replace the placeholder with the actual name of the DLT pipeline.
In your DLT pipeline configuration, add a parameter like this:

 

ExampleJSON :{ "name": "my_dlt_pipeline", "parameters": { "dlt_pipeline_id": "{{dlt_pipeline.name}}" } }
Then, in your Spark code, you can access this parameter using spark.conf.get("dlt_pipeline_id").
 
Method 2: Using spark.conf.set in the DLT pipeline
Alternatively, you can use the spark.conf.set method to set the dlt_pipeline_id configuration parameter in your DLT pipeline.
In your DLT pipeline code, add the following line:

 

ExamplePython:
This sets the dlt_pipeline_id configuration parameter to the name of the current DLT pipeline.
 
Method 3: Using a widget
You can also use a widget to store the DLT pipeline ID and access it later.
In your DLT pipeline configuration, add a widget like this:

 

ExampleJSON: { "name": "my_dlt_pipeline", "widgets": { "dlt_pipeline_id": { "type": "string", "value": "{{dlt_pipeline.name}}" } } }
 
Then, in your Spark code, you can access the widget value using dbutils.widgets.get("dlt_pipeline_id").
All of these methods should work, but Method 1 is probably the most straightforward. Let me know if you have any further questions!
0 Kudos

ashraf1395
Honored Contributor

‎02-18-2025 06:18 AM

Hi @mourakshit , 
I tried all the three methods you mentioned . None of them worked

method_1 returne pipeline_name or pipelin_id as printed value : {{dlt_pipeline.name}} {{dlt_pipeline.id}} not the actual values

methond_2 returned not conf like spark.databricks.pipeline.name exists
method_3 same error like method_1 

It would be really helpfull if you can show a demo example or screenshot by using any of the methods will be really helpful.

View solution in original post

0 Kudos

CaptainJack
New Contributor III

‎10-28-2025 02:16 PM

Did someone was able to get pipeline_id programaticaly?

0 Kudos