cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Can I pass parameters to a Delta Live Table pipeline at running time?

jfvizoso
New Contributor II

I need to execute a DLT pipeline from a Job, and I would like to know if there is any way of passing a parameter.

I know you can have settings in the pipeline that you use in the DLT notebook, but it seems you can only assign values to them when creating the pipeline. What I would like is to specify a value each time the pipeline is executed. Is that possible?

Thank you all.

4 REPLIES 4

kfoster
Contributor

You can call the API to update the pipeline to change the value or inside your notebook, you can make a call to a Delta Table, SQL, etc... to return the value needing to be used.

Debayan
Esteemed Contributor III
Esteemed Contributor III

Hi, what kind of parameters are you trying to change at running time? Is this the one:

https://docs.databricks.com/workflows/delta-live-tables/delta-live-tables-configuration.html

(Delta Live Tables settings specify one or more notebooks that implement a pipeline and the parameters specifying how to run the pipeline in an environment, for example, development, staging, or production. Delta Live Tables settings are expressed as JSON and can be modified in the Delta Live Tables UI.)

Kaniz_Fatma
Community Manager
Community Manager

Hi @José Fernández Vizoso​ , We haven’t heard from you on the last response from @kfoster and @Debayan Mukherjee​ and I was checking back to see if their suggestions helped you.

Or else, If you have any solution, please do share that with the community as it can be helpful to others.

Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.

Mustafa_Kamal
New Contributor II

Hi @jfvizoso ,

I also have the same scenario, did you find any work around.

Thanks in advance.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!