topic Re: What are the options for "spark_conf.spark.databricks.cluster.profile"? in Data Engineering

What are the options for "spark_conf.spark.databricks.cluster.profile"?

LeoGaller — Mon, 29 Apr 2024 18:55:20 GMT

Hey guys, I'm trying to find what are the options we can pass to

spark_conf.spark.databricks.cluster.profile

I know looking around that some of the available configs are singleNode and serverless, but there are others?

Where is the documentation of it?

Thank you!

Re: What are the options for "spark_conf.spark.databricks.cluster.profile"?

msamson — Wed, 18 Sep 2024 17:02:30 GMT

So, the "Other Profiles" that you mention are not documented in the documentation that you linked to, and I haven't been able to find any list of the other possible values despite what I thought were pretty decent googling skills (GPT4 plus web search is also not seemingly able to find it). I'm basically just trying to find the value for multi-node. Single-node is singleNode, and multi-node is... something? 😕

Re: What are the options for "spark_conf.spark.databricks.cluster.profile"?

LeoGaller — Sat, 21 Sep 2024 21:16:17 GMT

Hi @Retired_mod thank you for the detailed response. But it still lacks the documentation about the options to be passed to the config.

As @msamson shared, there are no docs about the other options beyond SingleNode and Serverless. I've tried to look for it even on databricks github repo but with no sucess.

Can you share with us what are the options we can use? Reviewing old clusters creation JSON, that config is never used. Does it means that this config was replaced by any other like data_security_mode?

Thank you!

Re: What are the options for "spark_conf.spark.databricks.cluster.profile"?

msamson — Sat, 21 Sep 2024 21:48:39 GMT

So, small update (sorry, I meant to post a few days ago but totally forgot): at least for multi-node, there is no value. Meaning, if you set a value, it will be that value, but if you don't it will be multinode. So if you, like I was, are setting up cluster policies and you wanted to, e.g., set single node as default but still allow multi node, you'd set spark.databricks.cluster.profile as singleNode, but set it to be optional. Can't remember the syntax off the top of my head, but it's along the lines of isOptional: true. Maybe not what you're looking for exactly, but that's what I needed. Good luck!

Re: What are the options for "spark_conf.spark.databricks.cluster.profile"?

s3 — Wed, 03 Sep 2025 07:44:06 GMT

Recently I got stuck with the same issue. However, in the new view of the form/template to create a policy, you have and option to delete the setting "spark_conf.spark.databricks.cluster.profile" by clicking on the "bin" icon. Once you did that, you are actually going multinode. Please ensure you have "ResourceClass" set to "MultiNode" and enabled autoscaling with min and max range.

Re: What are the options for "spark_conf.spark.databricks.cluster.profile"?

LeoGallerDbx — Thu, 04 Dec 2025 19:06:56 GMT

Looking internally, I was able to find the following:

For single node mode: the config should be set to 'singleNode'

For standard mode: the config should NOT be set to 'singleNode'

For serverless mode: the config should be set to 'serverless'

So, in summary, the options are 'serverless', 'singleNode', or not set.

Cheers.