how to list all spark session config variables

[gpierard]

In databricks I can set a config variable at session level, but it is not found in the context variables:

spark.conf.set(f"dataset.bookstore", '123') #dataset_bookstore
spark.conf.get(f"dataset.bookstore")#123
scf = spark.sparkContext.getConf()
allc = scf.getAll()
scf.contains(f"dataset.bookstore") # False

I understand there is a difference between session and context-level config variables, how can I retrieve all session-level variables using spark.conf?

Note: all_session_vars = spark.conf.getAll()

returns

AttributeError: 'RuntimeConfig' object has no attribute 'getAll'

so it looks like a runtime-level config

[Davidbruner]

 wrote:

spark.conf.set(f"dataset.bookstore", '123') #dataset_bookstore
spark.conf.get(f"dataset.bookstore")#123
scf = spark.sparkContext.getConf()
allc = scf.getAll()
scf.contains(f"dataset.bookstore") # False

 spark.conf?  Mayo Clinic Patient Portal

Note: all_session_vars = spark.conf.getAll()

returns

AttributeError: 'RuntimeConfig' object has no attribute 'getAll'

so it looks like a runtime-level config

---

Hello,

In Databricks, you can set session-level configuration variables using spark.conf.set(), but these session-level variables are distinct from the context-level variables. While you can retrieve session-level variables using spark.conf.get(), you cannot directly retrieve all session-level variables using spark.conf.getAll().

session_conf = spark.sparkContext.getConf()
all_session_vars = [(key, session_conf.get(key)) for key in session_conf.getAll()]

# Now all_session_vars contains a list of tuples with session-level variables

[Kaniz_Fatma]

Hi @gpierard , In Databricks, you can set and get configuration variables at the session level using spark.conf.set() and spark.conf.get() respectively. However, there is no direct method like getAll() to retrieve all session-level variables using spark.conf.The confusion arises because there are two types of configurations in Spark: SparkConf and RuntimeConfig. SparkConf is used when we create a SparkContext or SparkSession, and it's immutable once the context is created. On the other hand, RuntimeConfig (accessed via spark.conf) is used to set runtime SQL options, and it can be modified after the SparkSession is created.Unfortunately, the getAll method is not available in RuntimeConfig (spark.conf), it's only available in SparkConf. So, you can't retrieve all session-level variables using spark.conf.However, you can get specific configuration values if you know the key for the configuration.

[gpierard · Accepted solution]

Turns out there is a way, see https://stackoverflow.com/questions/76986516/how-to-retrieve-all-spark-session-config-variables

[RyanHager]

A while back I think I found a way to get python to list all the config values.  I was not able to re-create it.  Just make one of your notebook code sections scala (first line) and use the second line:

%scala

(spark.conf.getAll).foreach(println)