cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

how to list all spark session config variables

gpierard
New Contributor III

In databricks I can set a config variable at session level, but it is not found in the context variables:

spark.conf.set(f"dataset.bookstore", '123') #dataset_bookstore
spark.conf.get(f"dataset.bookstore")#123
scf = spark.sparkContext.getConf()
allc = scf.getAll()
scf.contains(f"dataset.bookstore") # False

I understand there is a difference between session and context-level config variables, how can I retrieve all session-level variables using spark.conf?

Note: all_session_vars = spark.conf.getAll()

returns

AttributeError: 'RuntimeConfig' object has no attribute 'getAll'

so it looks like a runtime-level config

1 ACCEPTED SOLUTION

Accepted Solutions

gpierard
New Contributor III
3 REPLIES 3

Davidbruner
New Contributor II

 wrote:

spark.conf.set(f"dataset.bookstore", '123') #dataset_bookstore
spark.conf.get(f"dataset.bookstore")#123
scf = spark.sparkContext.getConf()
allc = scf.getAll()
scf.contains(f"dataset.bookstore") # False

 spark.conf?  Mayo Clinic Patient Portal

Note: all_session_vars = spark.conf.getAll()

returns

AttributeError: 'RuntimeConfig' object has no attribute 'getAll'

so it looks like a runtime-level config


Hello,

In Databricks, you can set session-level configuration variables using spark.conf.set(), but these session-level variables are distinct from the context-level variables. While you can retrieve session-level variables using spark.conf.get(), you cannot directly retrieve all session-level variables using spark.conf.getAll().

session_conf = spark.sparkContext.getConf()
all_session_vars = [(key, session_conf.get(key)) for key in session_conf.getAll()]

# Now all_session_vars contains a list of tuples with session-level variables

 

gpierard
New Contributor III

RyanHager
Contributor

A while back I think I found a way to get python to list all the config values.  I was not able to re-create it.  Just make one of your notebook code sections scala (first line) and use the second line:

 

%scala
(spark.conf.getAll).foreach(println)

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group