DatabricksSession and SparkConf

fdeba · ‎07-25-2024

Hi,

I want to initialize a Spark session using `DatabricksSession`. However, it seems not possible to call `.config()` and pass it a `SparkConf` instance. The following works:

# Initialize the configuration for the Spark session
confSettings = [
    ("spark.sql.legacy.timeParserPolicy", "CORRECTED"),
    ("spark.sql.mapKeyDedupPolicy", "LAST_WIN"),
    ("spark.sql.legacy.parquet.nanosAsLong", "true"),
]
conf = SparkConf() \
    .setMaster("local") \
    .setAll(confSettings) \
    .setExecutorEnv(confSettings)

# Initialize a Spark session
spark = SparkSession.builder \
    .config(conf=conf) \
    .getOrCreate()

But the following throws an error:

```

# Initialize the configuration for the Spark session

confSettings = [

("spark.sql.legacy.timeParserPolicy", "CORRECTED"),

("spark.sql.mapKeyDedupPolicy", "LAST_WIN"),

("spark.sql.legacy.parquet.nanosAsLong", "true"),

]

conf = SparkConf() \

.setMaster("local") \

.setAll(confSettings) \

.setExecutorEnv(confSettings)

# Initialize a Spark session

spark = DatabricksSession.builder \

.profile("<profile-name>") \

.config(conf=conf) \

.getOrCreate()

```

Is there another way to set configuration for a Spark session when using `DatabricksSession`?

Thanks.

szymon_dybczak · ‎07-25-2024

Hi @fdeba ,

Maybe try to set them up after you acquire spark session with spark.conf.set , for example using loop:

# Initialize the configuration for the Spark session
confSettings = [
    ("spark.sql.legacy.timeParserPolicy", "CORRECTED"),
    ("spark.sql.mapKeyDedupPolicy", "LAST_WIN"),
    ("spark.sql.legacy.parquet.nanosAsLong", "true"),
]


# Initialize a Spark session
spark = DatabricksSession.builder \
    .profile("<profile-name>") \
    .config(conf=conf) \    
    .getOrCreate()

for conf in confSettings:
    spark.conf.set(conf[0], conf[1])

Witold · ‎07-25-2024

In almost all cases you don't need to create a new spark session, as Databricks will do it for you automatically.

If it's only about spark configurations, there are multiple ways to set it: