topic Parameterizing DLT Pipelines in Get Started Discussions

Parameterizing DLT Pipelines

Mustafa_Kamal — Fri, 19 Apr 2024 23:53:24 GMT

Hi Everyone,

I have DLTP pipeline which I need to execute for difference source systems. Need advise on how to parametrize this.

I have gone through many articles on the web, but it seems there is no accurate information available.

Can anyone please help.

Thanks,

Re: Parameterizing DLT Pipelines

AmanSehgal — Sat, 20 Apr 2024 00:25:42 GMT

You can provide parameters in the configuration section of DLT pipeline and access it in your code using spark.conf.get(<parameter_name>).

Parameterize DLT pipelines

Re: Parameterizing DLT Pipelines

Mustafa_Kamal — Sat, 20 Apr 2024 12:45:19 GMT

Thank you @AmanSehgal ,

I have done that and was able to execute the pipeline successfully. Bu t I need to change the parameter value at run time, so that the same pipeline can be used for multiple sources.

Can we pass parameters from Job to DLT Pipeline ?

Re: Parameterizing DLT Pipelines

AmanSehgal — Sun, 21 Apr 2024 11:59:47 GMT

I tried but it doesn't seem to be working. I tried passing job parameters to cluster configuration as {{job.paramters.name}} but it didn't pickup.

You might have to put a wrapper logic around your pipeline for now to parameterize it. For instance, write some text into a file in mounted location and then read it in the DLT pipeline for your conditional logic.

Re: Parameterizing DLT Pipelines

Mustafa_Kamal — Mon, 22 Apr 2024 07:15:50 GMT

Hi @AmanSehgal , Yes, thats true, I had tried different options here. Thank you so much for taking it time to investigate this.