How to capture dlt pipeline id / name using dynamic value reference

[ashraf1395]

Hi there,

I have a usecase where I want to set the dlt pipeline id in the configuration parameters of that dlt pipeline.

The way we can use workspace ids or task id in notebook task task_id = {{task.id}}/ {{task.name}} and can save them as parameters and can call later as dbutils.widgets.get("task_id").

Can we do something similar in dlt pipeline.

like dlt_pipeline_id = {{dlt_pipeline.name}} in the configurations and then use it spark.conf.get("dlt_pipeline_id").

or is there any other way to achieve this

[ashraf1395]

Any ideas on this @VZLA, @Alberto_Umana 

[mourakshit]

The answer is yes, you can achieve this in a DLT (Delta Live Tables) pipeline. Here are a few ways to do it:

Method 1: Using {{dlt_pipeline.name}} in the configuration

You can use the {{dlt_pipeline.name}} syntax in your DLT pipeline configuration, just like you would in a notebook task. This will replace the placeholder with the actual name of the DLT pipeline.

In your DLT pipeline configuration, add a parameter like this:

ExampleJSON :{ "name": "my_dlt_pipeline", "parameters": { "dlt_pipeline_id": "{{dlt_pipeline.name}}" } }

Then, in your Spark code, you can access this parameter using spark.conf.get("dlt_pipeline_id").

Method 2: Using spark.conf.set in the DLT pipeline

Alternatively, you can use the spark.conf.set method to set the dlt_pipeline_id configuration parameter in your DLT pipeline.

In your DLT pipeline code, add the following line:

ExamplePython:

spark.conf.set("dlt_pipeline_id", spark.conf.get("spark.databricks.pipeline.name"))

This sets the dlt_pipeline_id configuration parameter to the name of the current DLT pipeline.

Method 3: Using a widget

You can also use a widget to store the DLT pipeline ID and access it later.

In your DLT pipeline configuration, add a widget like this:

ExampleJSON: { "name": "my_dlt_pipeline", "widgets": { "dlt_pipeline_id": { "type": "string", "value": "{{dlt_pipeline.name}}" } } }

Then, in your Spark code, you can access the widget value using dbutils.widgets.get("dlt_pipeline_id").

All of these methods should work, but Method 1 is probably the most straightforward. Let me know if you have any further questions!

[ashraf1395 · Accepted solution]

Hi @mourakshit , 
I tried all the three methods you mentioned . None of them worked

method_1 returne pipeline_name or pipelin_id as printed value : {{dlt_pipeline.name}} {{dlt_pipeline.id}} not the actual values

methond_2 returned not conf like spark.databricks.pipeline.name exists
method_3 same error like method_1 

It would be really helpfull if you can show a demo example or screenshot by using any of the methods will be really helpful.