Help with getting photon usage

[alpacas]

I want to get whether photon was used for a job or not. The api lets me get this for maybe 40% of jobs through the runtime_engine field, but the majority of jobs are unspecified. How do I get whether photon was used for those cases? The docs mention looking at spark_version, but Im not sure what exactly within spark version indicates whether photon was used. Would appreciate help 

[Kaniz_Fatma]

Hi @alpacas, Determining whether Photon was used for a Spark job involves examining specific indicators.

Let’s break it down:

1. Photon Acceleration:

   • Photon is a high-performance runtime component in Databricks that accelerates SQL and DataFrame API calls. It enhances performance and reduces costs.
   • By default, Photon is enabled on compute running Databricks Runtime 9.1 LTS and above¹².
   • To check if Photon was used for a specific job, you can verify whether the job was executed on a compatible runtime version (9.1 LTS or higher).

2. Spark Version:

   • The spark_version field in your job metadata provides information about the Spark version used.
   • While the documentation doesn’t explicitly mention Photon, you can infer its usage based on the Spark version.
   • Here’s what to look for:
     • If the spark_version corresponds to Databricks Runtime 9.1 LTS or later, Photon acceleration is likely enabled.
     • If the spark_version is below 9.1 LTS, Photon may not be in use.
   • Remember that Photon is compatible with Apache Spark APIs, so no code changes are required².

3. Runtime Engine Field:

   • You mentioned that the runtime_engine field provides information for only 40% of jobs.
   • For the remaining unspecified jobs, focus on the spark_version.
   • If the spark_version aligns with the compatible range, assume Photon acceleration.

4. Best Practice:

   • To ensure accurate detection, consider using both the runtime_engine and spark_version fields.
   • If Photon is critical for your use case, explicitly specify the runtime version when submitting jobs.

Remember that while Photon is a powerful optimization, it’s essential to verify its usage based on the runtime environment.