How do I access to DLT advanced configuration from python notebook?

Chinu
Contributor

‎09-13-2023 12:19 PM

Hi Team, 
Im trying to get DLT Advanced Configuration value from the python dlt notebook. 

For example, I set "something": "some path" in Advanced configuration in DLT and I want to get the value from my dlt notebook. I tried "dbutils.widgets.get("something") in my notebook and triggered the pipeline but I got an error. 

How can I get the value? 

 

0 Kudos

jose_gonzalez
Databricks Employee
Databricks Employee

‎04-18-2024 03:02 PM

The following docs will help. Please check the examples https://docs.databricks.com/en/delta-live-tables/settings.html#parameterize-pipelines

Nastia
New Contributor III
In response to jose_gonzalez

‎06-18-2024 07:13 AM - edited ‎06-19-2024 01:24 AM

Hi @jose_gonzalez , I am getting the same error as the author of the post. in this example what "mypipeline" is referring to?

because I am getting the following error: 

org.apache.spark.SparkNoSuchElementException: [SQL_CONF_NOT_FOUND] The SQL config "mypipeline.host" cannot be found. Please verify that the config exists.

EDIT: I have figured out. To use advanced configuration value we just need to use it as following:

destination = spark.conf.get("host")

Mo
Databricks Employee
Databricks Employee

‎06-19-2024 07:32 AM

here you can find the documentation on how to use parameters in dlt (sql and python):

https://docs.databricks.com/en/delta-live-tables/settings.html#parameterize-dataset-declarations-in-...

0 Kudos