mourakshit
Databricks Employee
Databricks Employee

‎02-13-2025 11:02 AM

 

The answer is yes, you can achieve this in a DLT (Delta Live Tables) pipeline. Here are a few ways to do it:
Method 1: Using {{dlt_pipeline.name}} in the configuration
You can use the {{dlt_pipeline.name}} syntax in your DLT pipeline configuration, just like you would in a notebook task. This will replace the placeholder with the actual name of the DLT pipeline.
In your DLT pipeline configuration, add a parameter like this:

 

ExampleJSON :{ "name": "my_dlt_pipeline", "parameters": { "dlt_pipeline_id": "{{dlt_pipeline.name}}" } }
Then, in your Spark code, you can access this parameter using spark.conf.get("dlt_pipeline_id").
 
Method 2: Using spark.conf.set in the DLT pipeline
Alternatively, you can use the spark.conf.set method to set the dlt_pipeline_id configuration parameter in your DLT pipeline.
In your DLT pipeline code, add the following line:

 

ExamplePython:
This sets the dlt_pipeline_id configuration parameter to the name of the current DLT pipeline.
 
Method 3: Using a widget
You can also use a widget to store the DLT pipeline ID and access it later.
In your DLT pipeline configuration, add a widget like this:

 

ExampleJSON: { "name": "my_dlt_pipeline", "widgets": { "dlt_pipeline_id": { "type": "string", "value": "{{dlt_pipeline.name}}" } } }
 
Then, in your Spark code, you can access the widget value using dbutils.widgets.get("dlt_pipeline_id").
All of these methods should work, but Method 1 is probably the most straightforward. Let me know if you have any further questions!
0 Kudos