cancel
Showing results for 
Search instead for 
Did you mean: 
Community Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

Help with getting photon usage

alpacas
New Contributor

I want to get whether photon was used for a job or not. The api lets me get this for maybe 40% of jobs through the runtime_engine field, but the majority of jobs are unspecified. How do I get whether photon was used for those cases? The docs mention looking at spark_version, but Im not sure what exactly within spark version indicates whether photon was used. Would appreciate help 

1 REPLY 1

Kaniz
Community Manager
Community Manager

Hi @alpacasDetermining whether Photon was used for a Spark job involves examining specific indicators.

Let’s break it down:

  1. Photon Acceleration:

  2. Spark Version:

    • The spark_version field in your job metadata provides information about the Spark version used.
    • While the documentation doesn’t explicitly mention Photon, you can infer its usage based on the Spark version.
    • Here’s what to look for:
      • If the spark_version corresponds to Databricks Runtime 9.1 LTS or later, Photon acceleration is likely enabled.
      • If the spark_version is below 9.1 LTS, Photon may not be in use.
    • Remember that Photon is compatible with Apache Spark APIs, so no code changes are required2.
  3. Runtime Engine Field:

    • You mentioned that the runtime_engine field provides information for only 40% of jobs.
    • For the remaining unspecified jobs, focus on the spark_version.
    • If the spark_version aligns with the compatible range, assume Photon acceleration.
  4. Best Practice:

    • To ensure accurate detection, consider using both the runtime_engine and spark_version fields.
    • If Photon is critical for your use case, explicitly specify the runtime version when submitting jobs.

Remember that while Photon is a powerful optimization, it’s essential to verify its usage based on the runtime environment. 

 
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!