Cluster policy type

[thrinadhReddy]

Hi guys,

I am creating a cluster policy through json. 

"runtime_engine": {

"type": "fixed"

"value": "PHOTON"

}

When I run the above code... PHOTON option is getting enabled but graying out... What would I specify in type field so that the photon option should not be grayed out.

Thank you 😊

[Kaniz_Fatma]

Hi @thrinadhReddy, It appears that you’re trying to configure the runtime_engine in Databricks cluster policy using JSON. The PHOTON option is getting enabled but greyed out, and you’d like to know how to prevent that. Let’s dive into this!

1. What is Photon?

   • Photon is an acceleration technology that improves query performance in Databricks clusters. It leverages vectorized query execution and SIMD (Single Instruction, Multiple Data) instructions to speed up data processing.
   • By default, Photon is enabled on clusters running Databricks Runtime 9.1 LTS and above¹.
   • To manually enable or disable Photon on your cluster, you can select the “Use Photon Acceleration” checkbox when creating or editing the cluster. If you’re creating a cluster using the Clusters API, set the runtime_engine to PHOTON¹.

2. Grayed Out Option:

   • If the Photon option is grayed out, it might be due to certain limitations or compatibility issues.
   • For example, Photon acceleration is not supported for ML Runtimes². If you’re using an ML runtime, you won’t be able to enable Photon.
   • To avoid the grayed-out behavior, consider using a standard Databricks Runtime (above 9.1 LTS) and explicitly enable Photon.

3. Solution:

   • If you want to use Photon, ensure that you’re using a compatible runtime version (not an ML runtime) and follow these steps:
     • Create a cluster using Databricks Runtime (not ML Runtime).
     • Enable Photon by selecting the “Use Photon Acceleration” checkbox during cluster creation or editing.
     • Adjust worker and driver types as needed for optimal performance.

Remember to check the runtime version and compatibility when configuring Photon in your cluster.

[thrinadhReddy]

Hi Kaniz,

Thanks for the reply.

Yes. This is exactly what I am looking for.

"It appears that you’re trying to configure the runtime_engine in Databricks cluster policy using JSON. The PHOTON option is getting enabled but greyed out, and you’d like to know how to prevent that. Let’s dive into this!"

I don't want it to be grayed out.. I just want it to be checked and accessible...

I used standard Databricks Runtime (11.3 LTS (Scala 2.12, Spark 3.3.0)).

Is there anything other than type=fixed, so that it won't gray out.

[Yeshwanth]

@thrinadhReddy

This section includes a reference for each of the available policy types. There are two categories of policy types: fixed policies and limiting policies.

Fixed policies prevent user configuration on an attribute. The two types of fixed policies are:

https://docs.databricks.com/en/admin/clusters/policy-definition.html#supported-policy-types

Beast regards,

Yesh

[thrinadhReddy]

Hi Yeshwant,

I did go through that link. No combination of choices work in my scenario. I tried all type options, nothing works.