04-19-2023 09:05 AM
Hi Community,
I have successfully run a job through the API but would need to be able to pass parameters (configuration) to the DLT workflow via the API
I have tried passing JSON in this format:
{
"full_refresh": "true",
"configuration": [
{
"config1": "config1_value",
"config2": "config2_value"
}
]
}
The API seems happy with the structure of the JSON but config1 and config2 are not being overridden
Any help greatly appreciated.
04-28-2023 01:26 AM
Hi @Kaniz Fatma,
Just wondered if there was any update on this. This is quite an important aspect of how we would implement DLT pipelines so would be good to know if it can be done, or if it's coming.
Many thanks.
04-20-2023 01:34 AM
Hi @Kaniz Fatma, thanks for the detailed reply. Looks like the response is talking about a job, not a delta live tables pipeline. Apologies if my initial question was not clear enough...
I am using the Delta Live Tables API:
Delta Live Tables API guide - Azure Databricks | Microsoft Learn
And want to refresh a DLT pipeline... I can initiate a refresh but but need to be able to override the configuration of the DLT pipeline with the values I supply.
I am using Azure Data Factory to call the API so just need to know what the JSON format needs to be in the request body so I can override the parameters
08-23-2023 06:57 AM
@labromb - Please let me know if you found a solution to your problem. I'm trying to do the same thing.
08-24-2023 07:12 AM
@Mo - I tried the same thing but i do not see the DLT pipeline config being overriden. The Web activity in ADF has the below config:
and the DLT pipeline config has
Please let me know what I'm doing wrong.
08-24-2023 08:24 AM
@Mo - Thank you. I will try this out now and keep you posted. I am directly passing the parameters in the body of the Web Activity with the POST command. I will try the update and let you know.
Thank you so much for the suggestion.
04-04-2024 09:49 PM
@Manjula_Ganesap Can you let us know what you did to get this working? I have a similar use case.
08-18-2024 08:41 PM
@Manjula_Ganesap how exactly did you format the body in the ADF Web activity? it isn't working for me, I tried with the configuration json object, and also with the parameter directly, but the configuration value isn't overwritten in the dlt pipeline execution.
Only difference in my use case is that the dlt Advanced setting has more than one configuration variable and I only need to overwrite one.
Should I add all of them to the API call body to get one of them overwritten?
08-24-2023 09:16 AM
@Mo - it worked. Thank you so much.
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group