Parametrize the DLT pipeline for dynamic loading of many tables
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-21-2024 10:53 AM
I am trying to ingest hundreds of tables with CDC, where I want to create a generic/dynamic pipeline which can accept parameters (e.g table_name, schema, file path) and run the logic on it. However, I am not able to find a way to pass parameters to pipeline.
PS: I am aware of using the metadata table to iterate however I might trigger the pipeline using the REST API by passing parameters to it.
Also, I cannot use pipeline configurations for this as they can't be passed dynamically before triggering execution.
Labels:
- Labels:
-
Workflows