cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

when to activate photon and when not to ?

Enrique1987
New Contributor III

Photon appears as an option to check and uncheck as appropriate.


The use of Photon leads to higher consumption of DBUs and higher costs.
At what point does it pay off and when not to enable it.
More costs for the use of photon, but at the same time less computation time would incur less costs in the end.

How do you calculate when to use it and when not to use it? Do you have to go case by case seeing how much it costs activated and not activated or is there some other trick to take into account.

which is more cost effective in economic terms, if you need more computing power, increase the size of your cluster or enable photon?

1 ACCEPTED SOLUTION

Accepted Solutions

jacovangelder
Honored Contributor

This is my own experience: For SQL workloads, with not too many joins, it will speed things up. For building facts and dimensions using many joins, I found Photon to increase costs by a lot, while not bringing much better performance. 

The only real way of determining if it will benefit you is by comparing processing times with photon enabled and photon disabled. If the speed is not at least 2x faster, I would not recommend it, as it doubles the DBUs. 

View solution in original post

1 REPLY 1

jacovangelder
Honored Contributor

This is my own experience: For SQL workloads, with not too many joins, it will speed things up. For building facts and dimensions using many joins, I found Photon to increase costs by a lot, while not bringing much better performance. 

The only real way of determining if it will benefit you is by comparing processing times with photon enabled and photon disabled. If the speed is not at least 2x faster, I would not recommend it, as it doubles the DBUs. 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group