What are the options for "spark_conf.spark.databricks.cluster.profile"?

[LeoGaller]

Hey guys, I'm trying to find what are the options we can pass to 

spark_conf.spark.databricks.cluster.profile

I know looking around that some of the available configs are singleNode and serverless, but there are others?

Where is the documentation of it?

Thank you!

[Kaniz]

Hi @LeoGaller , The spark_conf.spark.databricks.cluster.profile configuration in Databricks allows you to specify the profile for a cluster.

Let’s explore the available options and where you can find the documentation.

1. Available Profiles:

   • SingleNode: This profile sets up a single-node cluster.
   • Serverless: The serverless profile is designed for running ad-hoc queries and jobs without the need to create a dedicated cluster. It automatically provisions resources as needed.
   • Other Profiles: While SingleNode and Serverless are commonly used, there are additional profiles that you can configure based on your requirements.

2. Documentation:

   • You can find detailed information about compute policies and profiles in the Databricks documentation. This article provides a reference for computing policy definitions, including available policy attributes and limitation types. It also includes sample policies for common use cases.

3. Policy Definitions:

   • Policy definitions are expressed in JSON format and allow you to set rules for various attributes controlled by the Clusters API.
   • For example, you can define a default autotermination time, restrict users from using pools, or enforce the use of specific runtime engines.
   • The supported attributes cover a wide range of settings, including autoscaling, autotermination, AWS attributes, cluster log configuration, and more.

If you have any more questions or need further assistance, feel free to ask!