Triggering DLT Pipelines with Dynamic Parameters

[Phani1]

Hi Team,

We have a scenario where we need to pass a dynamic parameter to a Spark job that will trigger a DLT pipeline in append mode. Can you please suggest an approach for this?

Regards,

Phani

[koji_kawamura]

Hi @Phani1 

DLT pipeline only support static parameters that we can define in the pipeline configuration. Would you elaborate your scenario? What parameters do you want to set dynamically?

[Phani1]

I want to trigger a Delta Live Tables (DLT) pipeline from a Databricks Job and pass a dynamic input parameter to apply a filter. However, it seems that pipeline settings can only be defined when creating the pipeline, and not when executing it. Is there a way to pass a dynamic value to the pipeline each time it's run?

[koji_kawamura]

Thanks for adding more details. IMHO, DLT pipelines are not designed to change it's behavior based on a dynamic value. It's more for keep doing the same thing over and over, from the last execution point incrementally. Stateful data processing.

Please let me try to imagine a possible situation. Let's say I have 3 different data sources, but the data ingestion and processing are nearly identical. So I'd like to call the same DLT pipeline 3 times from a workflow job, by passing a dynamic parameter pointing to different source locations, to reuse the same implementation.

In that case, I'd just write a DLT pipeline definition in a notebook. Create 3 DLT pipelines using DLT parameters to specify different source locations. Then execute the pipelines form a job.

Also, if you have a lot of ingestion routes and want to mass produce pipelines, Python meta programing approach may be helpful.

I hope I understand your point correctly.

[jericksoncea]

Workflow jobs implementing DLT pipelines seem to work differently than other jobs (notebook, dbt, etc) in terms of parameters...

For notebooks, job parameters are pushed down to the notebook parameters (widgets), overwriting the parameter defaults. A common scenario is persisting the job run ID in a column in the tables for records that are inserted or updated. This is a need I have.

Generally, pushing job parameters to DLT pipelines seems to be a common request (see Can I pass parameters to a Delta Live Table pipeli... - Databricks Community - 30440).

It is confusing and inconsistent that this cannot happen with DLT pipelines, i.e. jobs triggering the pipeline do not push parameters down to the DLT pipeline. 

Are there work arounds?

[jericksoncea]

I ended up using the Databricks SDK pipelines update before each run (a hack), to update the configuration. 

I think the DLT side of Databricks is its own world; the jobs and repository configuration works differently than other features, and it has only just been fully integrated with unity catalog (2025 DLT Update: Intelligent, fully governed data pipelines | Databricks Blog). It would be nice for some consistency...

[sas30]

found a working example -

databricks pipelines update <pipeline_id> --json @new_config.json

databricks pipelines start-update <Pipelineid>

where in use JSON for passing parameters.. every run update the parameters with new json file

[bombercorny]

can you provide some more details. Not really getting your answer...

[Sudharsan]

@koji_kawamura : I have more or less the same scenario say I have 3 tables.

The sources and targets are different but I would like to use a generic pipeline and pass in the source and target as a parameter and run them parallely.

@sas30 : can you be more elaborate

@bombercorny  : got any info on this??

[pradeep_singh]

If you’re looking to build a dynamic, configuration-driven DLT pipeline, a better approach is to use a configuration table. This table should include fields such as table_name, pipeline_name, table_properties, and other relevant settings. Your notebook can then query this table, applying filters for the table and pipeline names that are passed dynamically through variables. The resolved properties can then be accessed directly within your code.

You can always update the parameters and have things dynamic by updating this table . 

[pradeep_singh]

In your config table you can also set an table_active_status to set it to Y or N . If you want that tables in that pipeline set it to Y if you want to disable it for any reason use N . The code for that specific pipeline only run if its status is active .