Edthehead
Contributor III

You cannot pass parameters from a Databricks job to a DLT pipeline. Atleast not yet. You can see from the DLT rest API that there is no option for it to accept any parameters.

But there is a workaround.

But there is a workaround.

With the assumption that you are using ADF to trigger your job what you can try is first get the pipeline using the GET pipeline API.

Then use the update pipeline API to update the DLT pipeline with your parameters. This will be the configuration which you can see in the UI like the attached screenshot.

Edthehead_0-1733020738237.png

Of course you can have multiple parameters as per your use case.

Then trigger your Databricks job which will call your updated DLT pipeline.

Keep in mind, that even though concurrency of your job can be more than 1, only 1 instance of your DLT pipeline can run at a time. You will need to wait until it finishes before updating and running it again.