topic Re: How to pass configuration values to a Delta Live Tables job through the Delta Live Tables API in Data Engineering

How to pass configuration values to a Delta Live Tables job through the Delta Live Tables API

labromb — Wed, 19 Apr 2023 16:05:29 GMT

Hi Community,

I have successfully run a job through the API but would need to be able to pass parameters (configuration) to the DLT workflow via the API

I have tried passing JSON in this format:

{ 
    "full_refresh": "true",
    "configuration": [ 
        {
               "config1": "config1_value",
               "config2": "config2_value"
          }
    ]
}

The API seems happy with the structure of the JSON but config1 and config2 are not being overridden

Any help greatly appreciated.

Re: How to pass configuration values to a Delta Live Tables job through the Delta Live Tables API

labromb — Thu, 20 Apr 2023 08:34:26 GMT

Hi @Kaniz Fatma, thanks for the detailed reply. Looks like the response is talking about a job, not a delta live tables pipeline. Apologies if my initial question was not clear enough...

I am using the Delta Live Tables API:

Delta Live Tables API guide - Azure Databricks | Microsoft Learn

And want to refresh a DLT pipeline... I can initiate a refresh but but need to be able to override the configuration of the DLT pipeline with the values I supply.

I am using Azure Data Factory to call the API so just need to know what the JSON format needs to be in the request body so I can override the parameters

Re: How to pass configuration values to a Delta Live Tables job through the Delta Live Tables API

labromb — Fri, 28 Apr 2023 08:26:39 GMT

Hi @Kaniz Fatma,

Just wondered if there was any update on this. This is quite an important aspect of how we would implement DLT pipelines so would be good to know if it can be done, or if it's coming.

Many thanks.

Re: How to pass configuration values to a Delta Live Tables job through the Delta Live Tables API

Manjula_Ganesap — Wed, 23 Aug 2023 13:57:07 GMT

@labromb - Please let me know if you found a solution to your problem. I'm trying to do the same thing.

Re: How to pass configuration values to a Delta Live Tables job through the Delta Live Tables API

Manjula_Ganesap — Thu, 24 Aug 2023 14:12:46 GMT

@Mo - I tried the same thing but i do not see the DLT pipeline config being overriden. The Web activity in ADF has the below config:

and the DLT pipeline config has

Please let me know what I'm doing wrong.

Re: How to pass configuration values to a Delta Live Tables job through the Delta Live Tables API

Manjula_Ganesap — Thu, 24 Aug 2023 15:24:03 GMT

@Mo - Thank you. I will try this out now and keep you posted. I am directly passing the parameters in the body of the Web Activity with the POST command. I will try the update and let you know.

Thank you so much for the suggestion.

Re: How to pass configuration values to a Delta Live Tables job through the Delta Live Tables API

Manjula_Ganesap — Thu, 24 Aug 2023 16:16:36 GMT

@Mo - it worked. Thank you so much.

Re: How to pass configuration values to a Delta Live Tables job through the Delta Live Tables API

Edthehead — Fri, 05 Apr 2024 04:49:29 GMT

@Manjula_Ganesap Can you let us know what you did to get this working? I have a similar use case.

Re: How to pass configuration values to a Delta Live Tables job through the Delta Live Tables API

BLM — Mon, 19 Aug 2024 03:41:01 GMT

@Manjula_Ganesap how exactly did you format the body in the ADF Web activity? it isn't working for me, I tried with the configuration json object, and also with the parameter directly, but the configuration value isn't overwritten in the dlt pipeline execution.

Only difference in my use case is that the dlt Advanced setting has more than one configuration variable and I only need to overwrite one.

Should I add all of them to the API call body to get one of them overwritten?

Re: How to pass configuration values to a Delta Live Tables job through the Delta Live Tables API

nalindhar — Sat, 30 Nov 2024 15:54:08 GMT

Was anyone able to configure anything except full refresh true or false as override with api call for running a DLT. I have requirement where I need to dynamically pass config parameters, compute type, etc. Wondering if anyone was successfully able to resolve this. @Manjula_Ganesap I see your post saying it worked. Can you share what exactly you did?

Re: How to pass configuration values to a Delta Live Tables job through the Delta Live Tables API

Edthehead — Sun, 01 Dec 2024 02:39:18 GMT

You cannot pass parameters from a Databricks job to a DLT pipeline. Atleast not yet. You can see from the DLT rest API that there is no option for it to accept any parameters.

But there is a workaround.

With the assumption that you are using ADF to trigger your job what you can try is first get the pipeline using the GET pipeline API.

Then use the update pipeline API to update the DLT pipeline with your parameters. This will be the configuration which you can see in the UI like the attached screenshot.

Of course you can have multiple parameters as per your use case.

Then trigger your Databricks job which will call your updated DLT pipeline.

Keep in mind, that even though concurrency of your job can be more than 1, only 1 instance of your DLT pipeline can run at a time. You will need to wait until it finishes before updating and running it again.