topic How do I access to DLT advanced configuration from python notebook? in Data Engineering

How do I access to DLT advanced configuration from python notebook?

Chinu — Wed, 13 Sep 2023 19:19:44 GMT

Hi Team,
Im trying to get DLT Advanced Configuration value from the python dlt notebook.

For example, I set "something": "some path" in Advanced configuration in DLT and I want to get the value from my dlt notebook. I tried "dbutils.widgets.get("something") in my notebook and triggered the pipeline but I got an error.

How can I get the value?

Re: How do I access to DLT advanced configuration from python notebook?

jose_gonzalez — Thu, 18 Apr 2024 22:02:44 GMT

The following docs will help. Please check the examples https://docs.databricks.com/en/delta-live-tables/settings.html#parameterize-pipelines

Re: How do I access to DLT advanced configuration from python notebook?

Nastia — Wed, 19 Jun 2024 08:24:26 GMT

Hi @jose_gonzalez , I am getting the same error as the author of the post. in this example what "mypipeline" is referring to?

because I am getting the following error:

org.apache.spark.SparkNoSuchElementException: [SQL_CONF_NOT_FOUND] The SQL config "mypipeline.host" cannot be found. Please verify that the config exists.

EDIT: I have figured out. To use advanced configuration value we just need to use it as following:

destination = spark.conf.get("host")

Re: How do I access to DLT advanced configuration from python notebook?

Mo — Wed, 19 Jun 2024 14:32:06 GMT

here you can find the documentation on how to use parameters in dlt (sql and python):

https://docs.databricks.com/en/delta-live-tables/settings.html#parameterize-dataset-declarations-in-python-or-sql