wrote:
spark.conf.set(f"dataset.bookstore", '123') #dataset_bookstore
spark.conf.get(f"dataset.bookstore")#123
scf = spark.sparkContext.getConf()
allc = scf.getAll()
scf.contains(f"dataset.bookstore") # False
spark.conf? Mayo Clinic Patient Portal
Note: all_session_vars = spark.conf.getAll()
returns
AttributeError: 'RuntimeConfig' object has no attribute 'getAll'
so it looks like a runtime-level config
Hello,
In Databricks, you can set session-level configuration variables using spark.conf.set(), but these session-level variables are distinct from the context-level variables. While you can retrieve session-level variables using spark.conf.get(), you cannot directly retrieve all session-level variables using spark.conf.getAll().
session_conf = spark.sparkContext.getConf()
all_session_vars = [(key, session_conf.get(key)) for key in session_conf.getAll()]
# Now all_session_vars contains a list of tuples with session-level variables