How to make a DLT pipeline trigger another pipeline?

[guangyi]

For example, I have 2 DLT pipelines, one is used for computing user gender distribution, another is used for compute user location distribution.

In the first pipeline, I follow the medallion architecture creating the bronze, silver, gold table one by one.

In the second pipeline, considering it related to the user data also, I want reuse the silver table created in the first pipeline.

I implemented the pipeline code successfully. But I don’t know how to achieve this: when the silver table data been updated by the first pipeline, the second pipeline should be triggered as well for updating the data immediately to reflect the latest distribution 

[Kaniz_Fatma · Accepted solution]

Hi @guangyi, To trigger your second pipeline when the silver table data is updated by the first pipeline, you can follow these steps:

1. Event-Based Trigger:

   • On the first pipeline (the one creating the silver table), add a WebActivity at the end.
   • Configure this WebActivity to signal a custom event using an event hub. You can send an event to a custom event grid topic endpoint.
   • Create a custom event trigger in your second pipeline (the one that reuses the silver table). This trigger should listen for the event signal from the first pipeline.
   • When the event is received, it will trigger the second pipeline to run and update the data based on ...¹.

2. Azure Data Factory (ADF):

   • If you’re using Azure Data Factory, you can create a file in a storage account post successful completion of all triggers in the first pipeline.
   • Set up an event-based trigger in the second pipeline. This trigger will activate once the file becomes available in a blob (i.e., after the completion of past triggers).
   • This way, your second pipeline will be triggered based on the success of the first pipeline’s execut...².

If you have any further questions or need additional guidance, feel free to ask! 😊