topic Re: DatabricksSession and SparkConf in Data Engineering

DatabricksSession and SparkConf

fdeba — Thu, 25 Jul 2024 09:45:57 GMT

Hi,

I want to initialize a Spark session using `DatabricksSession`. However, it seems not possible to call `.config()` and pass it a `SparkConf` instance. The following works:

# Initialize the configuration for the Spark session confSettings = [ ("spark.sql.legacy.timeParserPolicy", "CORRECTED"), ("spark.sql.mapKeyDedupPolicy", "LAST_WIN"), ("spark.sql.legacy.parquet.nanosAsLong", "true"), ] conf = SparkConf() \ .setMaster("local") \ .setAll(confSettings) \ .setExecutorEnv(confSettings) # Initialize a Spark session spark = SparkSession.builder \ .config(conf=conf) \ .getOrCreate()

But the following throws an error:

```

# Initialize the configuration for the Spark session

confSettings = [

("spark.sql.legacy.timeParserPolicy", "CORRECTED"),

("spark.sql.mapKeyDedupPolicy", "LAST_WIN"),

("spark.sql.legacy.parquet.nanosAsLong", "true"),

]

conf = SparkConf() \

.setMaster("local") \

.setAll(confSettings) \

.setExecutorEnv(confSettings)

# Initialize a Spark session

spark = DatabricksSession.builder \

.profile("<profile-name>") \

.config(conf=conf) \

.getOrCreate()

```

Is there another way to set configuration for a Spark session when using `DatabricksSession`?

Thanks.

Re: DatabricksSession and SparkConf

szymon_dybczak — Thu, 25 Jul 2024 10:36:47 GMT

Hi @fdeba ,

Maybe try to set them up after you acquire spark session with spark.conf.set , for example using loop:

# Initialize the configuration for the Spark session confSettings = [ ("spark.sql.legacy.timeParserPolicy", "CORRECTED"), ("spark.sql.mapKeyDedupPolicy", "LAST_WIN"), ("spark.sql.legacy.parquet.nanosAsLong", "true"), ] # Initialize a Spark session spark = DatabricksSession.builder \ .profile("<profile-name>") \ .config(conf=conf) \ .getOrCreate() for conf in confSettings: spark.conf.set(conf[0], conf[1])

Re: DatabricksSession and SparkConf

Witold — Thu, 25 Jul 2024 10:27:06 GMT

In almost all cases you don't need to create a new spark session, as Databricks will do it for you automatically.

If it's only about spark configurations, there are multiple ways to set it: