Temporarily disable Photon

[AJMorgan591]

Is it possible to temporarily disable Photon?

I have a large workload that greatly benefits from Photon apart from a specific operation therein that is actually slowed by Photon. It's not worth creating a separate cluster for this operation however, so I'm wondering if it's possible to programmatically disable Photon (via Spark config or otherwise) and reenable it again later.

[Ryan_Chynoweth]

I do not believe this is possible. I believe photon is a specific runtime and not a configuration. I think you need to deploy different clusters to do that.

Are you sure it is slow due to photon? If there are computations not supported by photon it will fall back to Spark. You can see this in the execution plan. I believe that blue boxes are spark and yellow is photon.

It could be the case that the slow area of code is actually running on Spark. But validate that because if it is on photon you would want to know.

[AJMorgan591]

Thanks @Ryan Chynoweth​.

>> Are you sure it is slow due to photon?

Yup. It's not falling back to Spark, per the execution plan. In fact, removing Photon from the cluster (for testing purposes) speeds up this particular operation by about 2x. So Photon supports what I'm doing, but it's not as efficient at it as good ol' Spark. Moral of the story: Photon is good for some things, and potentially bad for others.

[karthik_p]

@Aaron Morgan​ you can disable photon by disabling "use photon acceleration" in cluster configuration.

please find attached screenshot

[Anonymous]

Hi @Aaron Morgan​ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!