Hi @jose_gonzalez , I am getting the same error as the author of the post. in this example what "mypipeline" is referring to?
because I am getting the following error:
org.apache.spark.SparkNoSuchElementException: [SQL_CONF_NOT_FOUND] The SQL config "mypipeline.host" cannot be found. Please verify that the config exists.
EDIT: I have figured out. To use advanced configuration value we just need to use it as following:
destination = spark.conf.get("host")