topic Re: Temporarily disable Photon in Data Engineering

Temporarily disable Photon

AJMorgan591 — Wed, 21 Sep 2022 20:29:50 GMT

Is it possible to temporarily disable Photon?

I have a large workload that greatly benefits from Photon apart from a specific operation therein that is actually slowed by Photon. It's not worth creating a separate cluster for this operation however, so I'm wondering if it's possible to programmatically disable Photon (via Spark config or otherwise) and reenable it again later.

Re: Temporarily disable Photon

Ryan_Chynoweth — Wed, 21 Sep 2022 22:16:25 GMT

I do not believe this is possible. I believe photon is a specific runtime and not a configuration. I think you need to deploy different clusters to do that.

Are you sure it is slow due to photon? If there are computations not supported by photon it will fall back to Spark. You can see this in the execution plan. I believe that blue boxes are spark and yellow is photon.

It could be the case that the slow area of code is actually running on Spark. But validate that because if it is on photon you would want to know.

Re: Temporarily disable Photon

AJMorgan591 — Thu, 22 Sep 2022 01:00:02 GMT

Thanks @Ryan Chynoweth.

>> Are you sure it is slow due to photon?

Yup. It's not falling back to Spark, per the execution plan. In fact, removing Photon from the cluster (for testing purposes) speeds up this particular operation by about 2x. So Photon supports what I'm doing, but it's not as efficient at it as good ol' Spark. Moral of the story: Photon is good for some things, and potentially bad for others.

Re: Temporarily disable Photon

karthik_p — Mon, 26 Sep 2022 20:03:51 GMT

@Aaron Morgan you can disable photon by disabling "use photon acceleration" in cluster configuration.

please find attached screenshot

Re: Temporarily disable Photon

Anonymous — Thu, 13 Oct 2022 08:26:24 GMT

Hi @Aaron Morgan

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!