โ03-13-2024 12:24 PM
I have started to use DLT in a prototype framework and I now face the below challenge for which any help would be appreciated.
First let me give a brief context:
โ03-14-2024 02:01 AM
Hi @MartinIsti, Passing parameters dynamically in a DLT (Delta Live Tables) pipeline can enhance flexibility and make your solution more robust.
While DLT doesnโt directly support task values for parameters, there are alternative approaches you can consider:
Configuration Parameters:
spark.conf.get("parameter_name", None)
syntax.Notebook Widgets:
dbutils.widgets.get("widget_name")
it to retrieve the value of a widget.REST API Calls:
{ "fullRefresh": true, "customParameter": "your_value" }
Airflow Integration:
DatabricksSubmitRunOperator
to submit a DLT pipeline run.โ03-14-2024 02:01 AM
Hi @MartinIsti, Passing parameters dynamically in a DLT (Delta Live Tables) pipeline can enhance flexibility and make your solution more robust.
While DLT doesnโt directly support task values for parameters, there are alternative approaches you can consider:
Configuration Parameters:
spark.conf.get("parameter_name", None)
syntax.Notebook Widgets:
dbutils.widgets.get("widget_name")
it to retrieve the value of a widget.REST API Calls:
{ "fullRefresh": true, "customParameter": "your_value" }
Airflow Integration:
DatabricksSubmitRunOperator
to submit a DLT pipeline run.โ03-17-2024 01:25 PM
Thanks Kaniz to your response. It would have been great to use a similar approach like the widgets to a normal notebook. Specifying these parameters at design time does not allow the flexibility needed for running my DLT pipeline truly metadata-driven.
I was also going towards using the job REST API from a notebook but then I ended up tweaking my configuration tables in a way that I can utilise a hardcoded parameter in the DLT definition and still have it dynamic.
If the REST API call functionality could be integrated into the workflows later on to pass these values as to other tasks, that would be really great!
I accept it as a solution because your third suggestion would work. I still keep hoping a more integrated approach will come in the future ๐
โ03-29-2024 12:00 PM
@Kaniz_Fatma Can you please provide some reference to REST API approach? I do not see that available on the docs.
TIA
Excited to expand your horizons with us? Click here to Register and begin your journey to success!
Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!