DatabricksSession and SparkConf

[fdeba]

Hi,

I want to initialize a Spark session using `DatabricksSession`. However, it seems not possible to call `.config()` and pass it a `SparkConf` instance. The following works:

# Initialize the configuration for the Spark session
confSettings = [
    ("spark.sql.legacy.timeParserPolicy", "CORRECTED"),
    ("spark.sql.mapKeyDedupPolicy", "LAST_WIN"),
    ("spark.sql.legacy.parquet.nanosAsLong", "true"),
]
conf = SparkConf() \
    .setMaster("local") \
    .setAll(confSettings) \
    .setExecutorEnv(confSettings)

# Initialize a Spark session
spark = SparkSession.builder \
    .config(conf=conf) \
    .getOrCreate()

But the following throws an error:

```

# Initialize the configuration for the Spark session

confSettings = [

    ("spark.sql.legacy.timeParserPolicy", "CORRECTED"),

    ("spark.sql.mapKeyDedupPolicy", "LAST_WIN"),

    ("spark.sql.legacy.parquet.nanosAsLong", "true"),

]

conf = SparkConf() \

    .setMaster("local") \

    .setAll(confSettings) \

    .setExecutorEnv(confSettings)

# Initialize a Spark session

spark = DatabricksSession.builder \

    .profile("<profile-name>") \

    .config(conf=conf) \    

    .getOrCreate()

```

Is there another way to set configuration for a Spark session when using `DatabricksSession`?

Thanks.

[Slash]

Hi @fdeba ,

Maybe try to set them up after you acquire spark session with spark.conf.set , for example using loop:

# Initialize the configuration for the Spark session
confSettings = [
    ("spark.sql.legacy.timeParserPolicy", "CORRECTED"),
    ("spark.sql.mapKeyDedupPolicy", "LAST_WIN"),
    ("spark.sql.legacy.parquet.nanosAsLong", "true"),
]


# Initialize a Spark session
spark = DatabricksSession.builder \
    .profile("<profile-name>") \
    .config(conf=conf) \    
    .getOrCreate()

for conf in confSettings:
    spark.conf.set(conf[0], conf[1])

[Witold]

In almost all cases you don't need to create a new spark session, as Databricks will do it for you automatically.

If it's only about spark configurations, there are multiple ways to set it:

• Cluster settings
• spark.conf.set