- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-04-2024 11:16 PM
Well, the reason why is probably because Photon is not standard Spark anymore. Databricks has completely rewritten the spark query engine in C++ (instead of Scala) and applied all kinds of optimizations to allow for faster processing.
So it is not merely a question of a new version of spark (which are all priced the same), using your Athena example, but actually another product which is integrated flawlessly in the Databricks platform.
If you do not want to pay extra for Photon, it is no problem at all to not use it.
IMO it seems fair to charge extra as the development of Photon was surely no easy and free task.
There is also the fact that if your job finishes faster you pay less, so that makes the pricing difference smaller.
In the end it is a question of figuring out if you want to use it for a certain workload or not.
I use it from time to time, but not always.