topic Re: How to capture dlt pipeline id / name using dynamic value reference in Data Engineering

How to capture dlt pipeline id / name using dynamic value reference

ashraf1395 — Tue, 11 Feb 2025 07:03:00 GMT

Hi there,

I have a usecase where I want to set the dlt pipeline id in the configuration parameters of that dlt pipeline.

The way we can use workspace ids or task id in notebook task task_id = {{task.id}}/ {{task.name}} and can save them as parameters and can call later as dbutils.widgets.get("task_id").

Can we do something similar in dlt pipeline.

like dlt_pipeline_id = {{dlt_pipeline.name}} in the configurations and then use it spark.conf.get("dlt_pipeline_id").

or is there any other way to achieve this

Re: How to capture dlt pipeline id / name using dynamic value reference

ashraf1395 — Thu, 13 Feb 2025 06:08:24 GMT

Any ideas on this @VZLA, @Alberto_Umana

Re: How to capture dlt pipeline id / name using dynamic value reference

mourakshit — Thu, 13 Feb 2025 19:02:58 GMT

The answer is yes, you can achieve this in a DLT (Delta Live Tables) pipeline. Here are a few ways to do it:

Method 1: Using {{dlt_pipeline.name}} in the configuration

You can use the {{dlt_pipeline.name}} syntax in your DLT pipeline configuration, just like you would in a notebook task. This will replace the placeholder with the actual name of the DLT pipeline.

In your DLT pipeline configuration, add a parameter like this:

ExampleJSON :{ "name": "my_dlt_pipeline", "parameters": { "dlt_pipeline_id": "{{dlt_pipeline.name}}" } }

Then, in your Spark code, you can access this parameter using spark.conf.get("dlt_pipeline_id").

Method 2: Using spark.conf.set in the DLT pipeline

Alternatively, you can use the spark.conf.set method to set the dlt_pipeline_id configuration parameter in your DLT pipeline.

In your DLT pipeline code, add the following line:

ExamplePython:

spark.conf.set("dlt_pipeline_id", spark.conf.get("spark.databricks.pipeline.name"))

This sets the dlt_pipeline_id configuration parameter to the name of the current DLT pipeline.

Method 3: Using a widget

You can also use a widget to store the DLT pipeline ID and access it later.

In your DLT pipeline configuration, add a widget like this:

ExampleJSON: { "name": "my_dlt_pipeline", "widgets": { "dlt_pipeline_id": { "type": "string", "value": "{{dlt_pipeline.name}}" } } }

Then, in your Spark code, you can access the widget value using dbutils.widgets.get("dlt_pipeline_id").

All of these methods should work, but Method 1 is probably the most straightforward. Let me know if you have any further questions!

Re: How to capture dlt pipeline id / name using dynamic value reference

ashraf1395 — Tue, 18 Feb 2025 14:18:22 GMT

Hi @mourakshit ,
I tried all the three methods you mentioned . None of them worked

method_1 returne pipeline_name or pipelin_id as printed value : {{dlt_pipeline.name}} {{dlt_pipeline.id}} not the actual values

methond_2 returned not conf like spark.databricks.pipeline.name exists
method_3 same error like method_1

It would be really helpfull if you can show a demo example or screenshot by using any of the methods will be really helpful.

Re: How to capture dlt pipeline id / name using dynamic value reference

CaptainJack — Tue, 28 Oct 2025 21:16:10 GMT

Did someone was able to get pipeline_id programaticaly?