cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

DLT - Get pipeline_id and update_id

Zeruno
New Contributor II

I need to insert pipeline_id and update_id in my Delta Live Table (DLT), the point being to know which pipeline created which row. How can I obtain this information?

I know you can get job_id and run_id from widgets but I don't know if these are the same things as pipeline_id and update_id.

I have a solution with event_hooks using Python but it looks wrong, I do not think it makes sense to listen for events in my DLT pipeline for this purpose.

#dlt

1 REPLY 1

szymon_dybczak
Contributor III

Hi @Zeruno ,

Those values are rather static. Maybe you can design process that as a first step will extract information from List Pipepline API and save them in delta table.
List pipelines | Pipelines API | REST API reference | Databricks on AWS
Than in your processing you lookup all needed information in this table.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group