cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

when to activate photon and when not to ?

Enrique1987
New Contributor

Photon appears as an option to check and uncheck as appropriate.


The use of Photon leads to higher consumption of DBUs and higher costs.
At what point does it pay off and when not to enable it.
More costs for the use of photon, but at the same time less computation time would incur less costs in the end.

How do you calculate when to use it and when not to use it? Do you have to go case by case seeing how much it costs activated and not activated or is there some other trick to take into account.

which is more cost effective in economic terms, if you need more computing power, increase the size of your cluster or enable photon?

1 ACCEPTED SOLUTION

Accepted Solutions

jacovangelder
Contributor III

This is my own experience: For SQL workloads, with not too many joins, it will speed things up. For building facts and dimensions using many joins, I found Photon to increase costs by a lot, while not bringing much better performance. 

The only real way of determining if it will benefit you is by comparing processing times with photon enabled and photon disabled. If the speed is not at least 2x faster, I would not recommend it, as it doubles the DBUs. 

View solution in original post

1 REPLY 1

jacovangelder
Contributor III

This is my own experience: For SQL workloads, with not too many joins, it will speed things up. For building facts and dimensions using many joins, I found Photon to increase costs by a lot, while not bringing much better performance. 

The only real way of determining if it will benefit you is by comparing processing times with photon enabled and photon disabled. If the speed is not at least 2x faster, I would not recommend it, as it doubles the DBUs. 

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!