https://community.databricks.com/t5/data-engineering/issue-pyspark-task-exception-handling-on-quot-shared-compute/m-p/86646#M37322 <P>Hi&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/118163">@geronimo_signol</a>&nbsp;,<BR /><BR />Recently, other user has reported similar behavior on shared clusters, and both issues seem to be related to Spark Connect.<BR />To verify whether your cluster is using Spark Connect, please run the following code in your notebook:</P><P>&nbsp;</P><LI-CODE lang="python">print(type(spark))</LI-CODE><P>&nbsp;</P><P>If the output shows &lt;class 'pyspark.sql.connect.session.SparkSession'&gt;, then your cluster is using Spark Connect. If it shows &lt;class 'pyspark.sql.session.SparkSession'&gt;, then it is normal spark.<BR />If it occurs on the cluster that uses Spark Connect, it mostly does not handle well schema updates.</P><P>Try the following:<BR />1.&nbsp;Verify Schema Before Transformation.&nbsp;Print the schema of the DataFrame to confirm that operator_lastname is present.</P><P>&nbsp;</P><LI-CODE lang="python">sdf = spark.table("`bronze_catalog`.`client_xyz`.`crew_data`") sdf.printSchema()</LI-CODE><P>&nbsp;</P><P>2.&nbsp;Check Column Existence.&nbsp;Before applying transformations, explicitly check if the column exists.</P><P>&nbsp;</P><LI-CODE lang="python">if 'operator_lastname' in sdf.columns: sdf = sdf.withColumn('transformed_failed', F.expr("UPPER(CAST(`operator_lastname` AS STRING))")) else: print("[ERROR] Column 'operator_lastname' does not exist in the DataFrame.")</LI-CODE><P>&nbsp;</P><P>3. Try to use caching to trigger schema refresh.</P><P>&nbsp;</P><LI-CODE lang="python">sdf = sdf.cache() # Refresh the DataFrame schema</LI-CODE><P>&nbsp;</P> Thu, 29 Aug 2024 18:49:31 GMT filipniziol 2024-08-29T18:49:31Z https://community.databricks.com/t5/data-engineering/issue-pyspark-task-exception-handling-on-quot-shared-compute/m-p/86368#M37315 <P><SPAN>I am experiencing an issue with a PySpark job that behaves differently depending on the compute environment in Databricks. And this is blocking us from deploying the job into the PROD environment for our planned release.</SPAN><BR /><BR /><SPAN>Specifically:</SPAN><BR /><BR /><SPAN>- When running the job on a personal cluster, everything works as expected. All exceptions within the try/catch blocks are successfully caught and handled.</SPAN><BR /><SPAN>- However, when I run the same job on a shared cluster, it fails, and no exceptions are being caught by the try/catch blocks.</SPAN></P><P><BR />Any guidance or insights you could provide would be greatly appreciated.</P><P>Example (running piece of code in a workspace notebook):&nbsp;<A href="https://github.com/user-attachments/assets/78b38c5a-98f6-4bb0-82c3-45946d6c5500" target="_blank" rel="noopener">https://github.com/user-attachments/assets/78b38c5a-98f6-4bb0-82c3-45946d6c5500</A></P><DIV class="">&nbsp;</DIV><P>Any ideas?&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/85537">@andrews</a>&nbsp;</P><P>&nbsp;</P> Thu, 29 Aug 2024 13:15:27 GMT https://community.databricks.com/t5/data-engineering/issue-pyspark-task-exception-handling-on-quot-shared-compute/m-p/86368#M37315 geronimo_signol 2024-08-29T13:15:27Z