topic Re: Photon plan invariant violated Error in Data Engineering

Photon plan invariant violated Error

Vishwanath_Rao — Wed, 06 Mar 2024 13:39:01 GMT

We've run into a niche error where we get the below message only on our non prod environment, with the same data, with the same code as our prod environment.

org.apache.spark.sql.execution.adaptive.InvalidAQEPlanException: Photon plan invariant violated: Broadcast joins have incompatible build plans regardingPhotonized vs. not-Photonized: ArrayBuffer(BroadcastHashJoin [COLUMN1#1372320], [COLUMN2#1422501], LeftOuter, BuildRight, false

Unsure as to what might be the issue, I've checked the spark logs, and it gives nothing on it as well, there doesn't seem to be any docs on these either. Any help is appreciated.

Re: Photon plan invariant violated Error

Vishwanath_Rao — Mon, 11 Mar 2024 14:57:22 GMT

Thank you @Retired_mod! It looks like the issue was with a recent release on the databricks end, I'd raised a support ticket just to be sure, the difference also died down when we set these two up, at the cluster level.

spark.conf.set("spark.sql.adaptive.enabled", "true")

spark.conf.set("spark.sql.optimizer.excludedRules", "")

Re: Photon plan invariant violated Error

JAC2703 — Thu, 07 Nov 2024 10:02:07 GMT

Hey, did you raise a ticket and get a resolution to this?