DLT - Get pipeline_id and update_id
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-07-2024 05:34 PM
I need to insert pipeline_id and update_id in my Delta Live Table (DLT), the point being to know which pipeline created which row. How can I obtain this information?
I know you can get job_id and run_id from widgets but I don't know if these are the same things as pipeline_id and update_id.
I have a solution with event_hooks using Python but it looks wrong, I do not think it makes sense to listen for events in my DLT pipeline for this purpose.
#dlt
- Labels:
-
Workflows
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-08-2024 12:16 AM
Hi @Zeruno ,
Those values are rather static. Maybe you can design process that as a first step will extract information from List Pipepline API and save them in delta table.
List pipelines | Pipelines API | REST API reference | Databricks on AWS
Than in your processing you lookup all needed information in this table.
