Does enabling Photon improve performance while lowering compute costs?

[neerajdubey_86]

Please refer screen shot and help in clarifying if enabling Photon lead to lower execution costs?

[balajij8]

@neerajdubey_86  Yes, enabling Photon generally improves performance while lowering compute costs. Photon is a native vectorized query engine that processes data in columnar batches and created by Databricks. It delivers better price performance for data and analytics compared to other cloud data warehouses with no code changes required. It accelerates SQL workloads, DataFrame operations, joins, shuffles, scans, and writes (including Delta, Iceberg and Parquet). Photon is already the default engine in all Databricks SQL warehouses.

The cost-performance benefit depends on few characteristics. Photon provides the greatest gains for longer-running queries on large datasets involving complex transformations such as joins, aggregations and wide-table operations. Queries that normally complete in under two seconds see minimal improvement because execution time is dominated by planning and scheduling overhead rather than data processing. Simple batch ETL jobs without wide transformations or large data volumes may see negligible impact. Additionally, Photon is enabled on a per-cluster or per-warehouse basis, and since it uses the same compute resources, the cost savings come from faster completion times rather than lower per-hour rates - so workloads that don't benefit from acceleration could see higher per-unit costs with no corresponding speedup. More details here

[Ashwin_DSA]

Hi @neerajdubey_86,

Photon will almost certainly make your queries faster, but whether it also makes them cheaper depends on your workload.

Photon is Databricks native vectorized query engine. It replaces the JVM-based Spark execution layer with a C++ runtime that processes data in columnar batches. That means no garbage collection pauses, no JIT warm-up, and the ability to take advantage of SIMD instructions on modern CPUs. For supported operations like scans, joins, aggregations, and writes, the performance improvement can be substantial. Databricks cites up to 5x better price/performance on TPC-DS benchmarks. And the best part is you don't need to change any code. It works with your existing SQL and DataFrame APIs.

The cost side is a bit more nuanced. Photon carries a higher DBU rate than standard Spark, so the cost savings come from the fact that your jobs finish faster and therefore consume fewer total compute hours. If a job runs 3x faster but the DBU rate is only 2x higher, you come out ahead on total cost. For SQL warehouses, this is a non-issue... Photon is the default engine at the same price, so it's a pure win. For batch jobs and pipelines, the math depends on how much of your workload actually runs in Photon. If most of your query plan is covered by Photon-supported operations, you will likely see both faster execution and lower total cost. But if your workload is heavy on UDFs, RDD APIs, or other unsupported operations, Photon falls back to Spark for those parts, and you end up paying the premium rate without getting the full speedup.

My recommendation would be to check how much of your query actually runs on Photon. You can do this via the Spark UI (Photon operators show in orange, Spark in blue) or the Query Profile in SQL warehouses (which shows the percentage of task time in Photon). If your Photon coverage is high.... say above 80%... you are in a great position for both performance and cost gains. Below 50%, you are unlikely to see meaningful savings and may actually pay more.

You can also refer to Databricks own cost optimization guide where it says the observed speedup can lead to significant cost savings, and jobs that run regularly should be evaluated to see whether they are not only faster but also cheaper with Photon. So the recommendation is to try it, measure the before-and-after on your recurring jobs, and keep it where the numbers work in your favour. For SQL warehouses, just leave it on. There is no downside.

Hope this helps.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.