Temporarily disable Photon
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-21-2022 01:29 PM
Is it possible to temporarily disable Photon?
I have a large workload that greatly benefits from Photon apart from a specific operation therein that is actually slowed by Photon. It's not worth creating a separate cluster for this operation however, so I'm wondering if it's possible to programmatically disable Photon (via Spark config or otherwise) and reenable it again later.
- Labels:
-
Photon
-
Spark config
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-21-2022 03:16 PM
I do not believe this is possible. I believe photon is a specific runtime and not a configuration. I think you need to deploy different clusters to do that.
Are you sure it is slow due to photon? If there are computations not supported by photon it will fall back to Spark. You can see this in the execution plan. I believe that blue boxes are spark and yellow is photon.
It could be the case that the slow area of code is actually running on Spark. But validate that because if it is on photon you would want to know.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-21-2022 06:00 PM
Thanks @Ryan Chynoweth.
>> Are you sure it is slow due to photon?
Yup. It's not falling back to Spark, per the execution plan. In fact, removing Photon from the cluster (for testing purposes) speeds up this particular operation by about 2x. So Photon supports what I'm doing, but it's not as efficient at it as good ol' Spark. Moral of the story: Photon is good for some things, and potentially bad for others.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-26-2022 01:03 PM
@Aaron Morgan you can disable photon by disabling "use photon acceleration" in cluster configuration.
please find attached screenshot
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-13-2022 01:26 AM
Hi @Aaron Morgan
Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.
We'd love to hear from you.
Thanks!