wrote:

spark.conf.set(f"dataset.bookstore", '123') #dataset_bookstore
spark.conf.get(f"dataset.bookstore")#123
scf = spark.sparkContext.getConf()
allc = scf.getAll()
scf.contains(f"dataset.bookstore") # False

 spark.conf?  Mayo Clinic Patient Portal

Note: all_session_vars = spark.conf.getAll()

returns

AttributeError: 'RuntimeConfig' object has no attribute 'getAll'

so it looks like a runtime-level config

Hello,

In Databricks, you can set session-level configuration variables using spark.conf.set(), but these session-level variables are distinct from the context-level variables. While you can retrieve session-level variables using spark.conf.get(), you cannot directly retrieve all session-level variables using spark.conf.getAll().

session_conf = spark.sparkContext.getConf()
all_session_vars = [(key, session_conf.get(key)) for key in session_conf.getAll()]

# Now all_session_vars contains a list of tuples with session-level variables

A while back I think I found a way to get python to list all the config values.  I was not able to re-create it.  Just make one of your notebook code sections scala (first line) and use the second line: