topic Re: Triggering DLT Pipelines with Dynamic Parameters in Data Engineering

Triggering DLT Pipelines with Dynamic Parameters

Phani1 — Mon, 03 Mar 2025 12:37:42 GMT

Hi Team,

We have a scenario where we need to pass a dynamic parameter to a Spark job that will trigger a DLT pipeline in append mode. Can you please suggest an approach for this?

Regards,

Phani

Re: Triggering DLT Pipelines with Dynamic Parameters

koji_kawamura — Tue, 04 Mar 2025 02:55:11 GMT

Hi @Phani1

DLT pipeline only support static parameters that we can define in the pipeline configuration. Would you elaborate your scenario? What parameters do you want to set dynamically?

Re: Triggering DLT Pipelines with Dynamic Parameters

Phani1 — Tue, 04 Mar 2025 08:34:18 GMT

I want to trigger a Delta Live Tables (DLT) pipeline from a Databricks Job and pass a dynamic input parameter to apply a filter. However, it seems that pipeline settings can only be defined when creating the pipeline, and not when executing it. Is there a way to pass a dynamic value to the pipeline each time it's run?

Re: Triggering DLT Pipelines with Dynamic Parameters

koji_kawamura — Wed, 05 Mar 2025 09:25:54 GMT

Thanks for adding more details. IMHO, DLT pipelines are not designed to change it's behavior based on a dynamic value. It's more for keep doing the same thing over and over, from the last execution point incrementally. Stateful data processing.

Please let me try to imagine a possible situation. Let's say I have 3 different data sources, but the data ingestion and processing are nearly identical. So I'd like to call the same DLT pipeline 3 times from a workflow job, by passing a dynamic parameter pointing to different source locations, to reuse the same implementation.

In that case, I'd just write a DLT pipeline definition in a notebook. Create 3 DLT pipelines using DLT parameters to specify different source locations. Then execute the pipelines form a job.

Also, if you have a lot of ingestion routes and want to mass produce pipelines, Python meta programing approach may be helpful.

I hope I understand your point correctly.

Re: Triggering DLT Pipelines with Dynamic Parameters

jericksoncea — Mon, 12 May 2025 08:52:36 GMT

Workflow jobs implementing DLT pipelines seem to work differently than other jobs (notebook, dbt, etc) in terms of parameters...

For notebooks, job parameters are pushed down to the notebook parameters (widgets), overwriting the parameter defaults. A common scenario is persisting the job run ID in a column in the tables for records that are inserted or updated. This is a need I have.

Generally, pushing job parameters to DLT pipelines seems to be a common request (see Can I pass parameters to a Delta Live Table pipeli... - Databricks Community - 30440).

It is confusing and inconsistent that this cannot happen with DLT pipelines, i.e. jobs triggering the pipeline do not push parameters down to the DLT pipeline.

Are there work arounds?

Re: Triggering DLT Pipelines with Dynamic Parameters

jericksoncea — Thu, 22 May 2025 07:26:50 GMT

I ended up using the Databricks SDK pipelines update before each run (a hack), to update the configuration.

I think the DLT side of Databricks is its own world; the jobs and repository configuration works differently than other features, and it has only just been fully integrated with unity catalog (2025 DLT Update: Intelligent, fully governed data pipelines | Databricks Blog). It would be nice for some consistency...

Re: Triggering DLT Pipelines with Dynamic Parameters

sas30 — Mon, 08 Sep 2025 22:54:56 GMT

found a working example -

databricks pipelines update <pipeline_id> --json @new_config.json

databricks pipelines start-update <Pipelineid>

where in use JSON for passing parameters.. every run update the parameters with new json file

Re: Triggering DLT Pipelines with Dynamic Parameters

bombercorny — Mon, 15 Dec 2025 10:36:06 GMT

can you provide some more details. Not really getting your answer...

Re: Triggering DLT Pipelines with Dynamic Parameters

Sudharsan — Fri, 09 Jan 2026 20:54:24 GMT

@koji_kawamura : I have more or less the same scenario say I have 3 tables.

The sources and targets are different but I would like to use a generic pipeline and pass in the source and target as a parameter and run them parallely.

@sas30 : can you be more elaborate

@bombercorny : got any info on this??

Re: Triggering DLT Pipelines with Dynamic Parameters

pradeep_singh — Sun, 01 Feb 2026 14:11:31 GMT

If you’re looking to build a dynamic, configuration-driven DLT pipeline, a better approach is to use a configuration table. This table should include fields such as table_name, pipeline_name, table_properties, and other relevant settings. Your notebook can then query this table, applying filters for the table and pipeline names that are passed dynamically through variables. The resolved properties can then be accessed directly within your code.

You can always update the parameters and have things dynamic by updating this table .

Re: Triggering DLT Pipelines with Dynamic Parameters

pradeep_singh — Sun, 01 Feb 2026 14:33:28 GMT

In your config table you can also set an table_active_status to set it to Y or N . If you want that tables in that pipeline set it to Y if you want to disable it for any reason use N . The code for that specific pipeline only run if its status is active .