How to pass configuration values to a Delta Live Tables job through the Delta Live Tables API

[labromb]

Hi Community,

I have successfully run a job through the API but would need to be able to pass parameters (configuration) to the DLT workflow via the API

I have tried passing JSON in this format:

{ 
    "full_refresh": "true",
    "configuration": [ 
        {
               "config1": "config1_value",
               "config2": "config2_value"
          }
    ]
}
 
 
 

The API seems happy with the structure of the JSON but config1 and config2 are not being overridden

Any help greatly appreciated.

[labromb]

Hi @Kaniz Fatma​,

Just wondered if there was any update on this. This is quite an important aspect of how we would implement DLT pipelines so would be good to know if it can be done, or if it's coming.

Many thanks.

[labromb]

Hi @Kaniz Fatma​, thanks for the detailed reply. Looks like the response is talking about a job, not a delta live tables pipeline. Apologies if my initial question was not clear enough...

I am using the Delta Live Tables API:

Delta Live Tables API guide - Azure Databricks | Microsoft Learn

And want to refresh a DLT pipeline... I can initiate a refresh but but need to be able to override the configuration of the DLT pipeline with the values I supply.

I am using Azure Data Factory to call the API so just need to know what the JSON format needs to be in the request body so I can override the parameters

[Manjula_Ganesap]

@labromb - Please let me know if you found a solution to your problem. I'm trying to do the same thing. 

[Manjula_Ganesap]

@Mo - I tried the same thing but i do not see the DLT pipeline config being overriden. The Web activity in ADF has the below config:

and the DLT pipeline config has 

Please let me know what I'm doing wrong. 

[Manjula_Ganesap]

@Mo  - Thank you. I will try this out now and keep you posted. I am directly passing the parameters in the body of the Web Activity with the POST command. I will try the update and let you know. 

Thank you so much for the suggestion. 

[Edthehead]

@Manjula_Ganesap  Can you let us know what you did to get this working? I have a similar use case.

[BLM]

@Manjula_Ganesap how exactly did you format the body in the ADF Web activity? it isn't working for me, I tried with the configuration json object, and also with the parameter directly, but the configuration value isn't overwritten in the dlt pipeline execution.

Only difference in my use case is that the dlt Advanced setting has more than one configuration variable and I only need to overwrite one.

Should I add all of them to the API call body to get one of them overwritten?

[Manjula_Ganesap]

@Mo - it worked. Thank you so much.