topic Re: Databricks Error while executing this line of code in Data Engineering

topic Re: Databricks Error while executing this line of code in Data Engineering https://community.databricks.com/t5/data-engineering/databricks-error-while-executing-this-line-of-code/m-p/62821#M32106 <P>Please suggest correct version of libraries i need to install to avoid above error.</P><P> </P><P> </P> Thu, 07 Mar 2024 08:14:55 GMT NT911 2024-03-07T08:14:55Z Databricks Error while executing this line of code https://community.databricks.com/t5/data-engineering/databricks-error-while-executing-this-line-of-code/m-p/62561#M32000 <DIV><DIV><SPAN>import</SPAN><SPAN> geopandas </SPAN><SPAN>as</SPAN><SPAN> gpd</SPAN></DIV><DIV><SPAN>from</SPAN><SPAN> shapely.geometry </SPAN><SPAN>import</SPAN> <SPAN>*</SPAN></DIV><DIV><SPAN>Pd_csv_sel_pq_gg </SPAN><SPAN>=</SPAN><SPAN> gpd.</SPAN><SPAN>GeoDataFrame</SPAN><SPAN>(Points_csv_sel_pq_gg.</SPAN><SPAN>toPandas</SPAN><SPAN>(), </SPAN><SPAN>geometry</SPAN><SPAN>=</SPAN><SPAN>"geometry"</SPAN><SPAN>)</SPAN></DIV><DIV> </DIV><DIV><SPAN>Error is given below </SPAN></DIV><DIV> </DIV><DIV><DIV>/databricks/spark/python/pyspark/sql/pandas/utils.py:37: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.</DIV><DIV>  if LooseVersion(pandas.__version__) < LooseVersion(minimum_pandas_version):</DIV><DIV>/databricks/spark/python/pyspark/sql/pandas/utils.py:64: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.</DIV><DIV>  if LooseVersion(pyarrow.__version__) < LooseVersion(minimum_pyarrow_version):</DIV><DIV>/databricks/spark/python/pyspark/sql/pandas/conversion.py:216: UserWarning: toPandas attempted Arrow optimization because 'spark.sql.execution.arrow.pyspark.enabled' is set to true, but has reached the error below and can not continue. Note that 'spark.sql.execution.arrow.pyspark.fallback.enabled' does not have an effect on failures in the middle of computation.</DIV><DIV>  An error occurred while calling o1101.getResult.</DIV><DIV>: org.apache.spark.SparkException: Exception thrown in awaitResult: Job aborted due to stage failure: Task 0 in stage 23.0 failed 4 times, most recent failure: Lost task 0.3 in stage 23.0 (TID 18) (10.139.64.11 executor 0): java.io.InvalidClassException: org.apache.spark.sql.sedona_sql.expressions.ST_FlipCoordinates; local class incompatible: stream classdesc serialVersionUID = -8818399722059471005, local class serialVersionUID = -572255679765828529</DIV><DIV><SPAN>at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:699)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2005)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1852)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2186)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readArray(ObjectInputStream.java:2119)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1657)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readArray(ObjectInputStream.java:2119)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1657)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readArray(ObjectInputStream.java:2119)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1657)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)</SPAN></DIV><DIV><SPAN>at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:527)</SPAN></DIV><DIV><SPAN>at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)</SPAN></DIV><DIV><SPAN>at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)</SPAN></DIV><DIV><SPAN>at java.lang.reflect.Method.invoke(Method.java:498)</SPAN></DIV><DIV><SPAN>at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1184)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2322)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)</SPAN></DIV><DIV><SPAN>at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:87)</SPAN></DIV><DIV><SPAN>at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:129)</SPAN></DIV><DIV><SPAN>at org.apache.spark.scheduler.ResultTask.$anonfun$runTask$2(ResultTask.scala:72)</SPAN></DIV><DIV><SPAN>at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)</SPAN></DIV><DIV><SPAN>at org.apache.spark.scheduler.ResultTask.$anonfun$runTask$1(ResultTask.scala:72)</SPAN></DIV><DIV><SPAN>at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)</SPAN></DIV><DIV><SPAN>at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)</SPAN></DIV><DIV><SPAN>at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:196)</SPAN></DIV><DIV><SPAN>at org.apache.spark.scheduler.Task.doRunTask(Task.scala:181)</SPAN></DIV><DIV><SPAN>at org.apache.spark.scheduler.Task.$anonfun$run$5(Task.scala:146)</SPAN></DIV><DIV><SPAN>at com.databricks.unity.EmptyHandle$.runWithAndClose(UCSHandle.scala:125)</SPAN></DIV><DIV><SPAN>at org.apache.spark.scheduler.Task.$anonfun$run$1(Task.scala:146)</SPAN></DIV><DIV><SPAN>at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)</SPAN></DIV><DIV><SPAN>at org.apache.spark.scheduler.Task.run(Task.scala:99)</SPAN></DIV><DIV><SPAN>at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$8(Executor.scala:899)</SPAN></DIV><DIV><SPAN>at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1709)</SPAN></DIV><DIV><SPAN>at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:902)</SPAN></DIV><DIV><SPAN>at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)</SPAN></DIV><DIV><SPAN>at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)</SPAN></DIV><DIV><SPAN>at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:797)</SPAN></DIV><DIV><SPAN>at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)</SPAN></DIV><DIV><SPAN>at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)</SPAN></DIV><DIV><SPAN>at java.lang.Thread.run(Thread.java:750)</SPAN></DIV><DIV> </DIV><DIV>Driver stacktrace:</DIV><DIV><SPAN>at org.apache.spark.util.SparkThreadUtils$.awaitResult(SparkThreadUtils.scala:54)</SPAN></DIV><DIV><SPAN>at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:462)</SPAN></DIV><DIV><SPAN>at org.apache.spark.security.SocketAuthServer.getResult(SocketAuthServer.scala:108)</SPAN></DIV><DIV><SPAN>at org.apache.spark.security.SocketAuthServer.getResult(SocketAuthServer.scala:104)</SPAN></DIV><DIV><SPAN>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)</SPAN></DIV><DIV><SPAN>at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)</SPAN></DIV><DIV><SPAN>at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)</SPAN></DIV><DIV><SPAN>at java.lang.reflect.Method.invoke(Method.java:498)</SPAN></DIV><DIV><SPAN>at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)</SPAN></DIV><DIV><SPAN>at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:397)</SPAN></DIV><DIV><SPAN>at py4j.Gateway.invoke(Gateway.java:306)</SPAN></DIV><DIV><SPAN>at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)</SPAN></DIV><DIV><SPAN>at py4j.commands.CallCommand.execute(CallCommand.java:79)</SPAN></DIV><DIV><SPAN>at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:195)</SPAN></DIV><DIV><SPAN>at py4j.ClientServerConnection.run(ClientServerConnection.java:115)</SPAN></DIV><DIV><SPAN>at java.lang.Thread.run(Thread.java:750)</SPAN></DIV><DIV>Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 23.0 failed 4 times, most recent failure: Lost task 0.3 in stage 23.0 (TID 18) (10.139.64.11 executor 0): java.io.InvalidClassException: org.apache.spark.sql.sedona_sql.expressions.ST_FlipCoordinates; local class incompatible: stream classdesc serialVersionUID = -8818399722059471005, local class serialVersionUID = -572255679765828529</DIV><DIV><SPAN>at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:699)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2005)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1852)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2186)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readArray(ObjectInputStream.java:2119)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1657)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readArray(ObjectInputStream.java:2119)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1657)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readArray(ObjectInputStream.java:2119)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1657)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)</SPAN></DIV><DIV><SPAN>at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:527)</SPAN></DIV><DIV><SPAN>at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)</SPAN></DIV><DIV><SPAN>at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)</SPAN></DIV><DIV><SPAN>at java.lang.reflect.Method.invoke(Method.java:498)</SPAN></DIV><DIV><SPAN>at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1184)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2322)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)</SPAN></DIV><DIV><SPAN>at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:87)</SPAN></DIV><DIV><SPAN>at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:129)</SPAN></DIV><DIV><SPAN>at org.apache.spark.scheduler.ResultTask.$anonfun$runTask$2(ResultTask.scala:72)</SPAN></DIV><DIV><SPAN>at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)</SPAN></DIV><DIV><SPAN>at org.apache.spark.scheduler.ResultTask.$anonfun$runTask$1(ResultTask.scala:72)</SPAN></DIV><DIV><SPAN>at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)</SPAN></DIV><DIV><SPAN>at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)</SPAN></DIV><DIV><SPAN>at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:196)</SPAN></DIV><DIV><SPAN>at org.apache.spark.scheduler.Task.doRunTask(Task.scala:181)</SPAN></DIV><DIV><SPAN>at org.apache.spark.scheduler.Task.$anonfun$run$5(Task.scala:146)</SPAN></DIV><DIV><SPAN>at com.databricks.unity.EmptyHandle$.runWithAndClose(UCSHandle.scala:125)</SPAN></DIV><DIV><SPAN>at org.apache.spark.scheduler.Task.$anonfun$run$1(Task.scala:146)</SPAN></DIV><DIV><SPAN>at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)</SPAN></DIV><DIV><SPAN>at org.apache.spark.scheduler.Task.run(Task.scala:99)</SPAN></DIV><DIV><SPAN>at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$8(Executor.scala:899)</SPAN></DIV><DIV><SPAN>at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1709)</SPAN></DIV><DIV><SPAN>at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:902)</SPAN></DIV><DIV><SPAN>at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)</SPAN></DIV><DIV><SPAN>at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)</SPAN></DIV><DIV><SPAN>at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:797)</SPAN></DIV><DIV><SPAN>at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)</SPAN></DIV><DIV><SPAN>at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)</SPAN></DIV><DIV><SPAN>at java.lang.Thread.run(Thread.java:750)</SPAN></DIV><DIV> </DIV><DIV>Driver stacktrace:</DIV><DIV><SPAN>at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:3628)</SPAN></DIV><DIV><SPAN>at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:3559)</SPAN></DIV><DIV><SPAN>at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:3546)</SPAN></DIV><DIV><SPAN>at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)</SPAN></DIV><DIV><SPAN>at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)</SPAN></DIV><DIV><SPAN>at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)</SPAN></DIV><DIV><SPAN>at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:3546)</SPAN></DIV><DIV><SPAN>at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1521)</SPAN></DIV><DIV><SPAN>at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1521)</SPAN></DIV><DIV><SPAN>at scala.Option.foreach(Option.scala:407)</SPAN></DIV><DIV><SPAN>at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1521)</SPAN></DIV><DIV><SPAN>at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:3875)</SPAN></DIV><DIV><SPAN>at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:3787)</SPAN></DIV><DIV><SPAN>at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:3775)</SPAN></DIV><DIV><SPAN>at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:51)</SPAN></DIV><DIV><SPAN>at org.apache.spark.scheduler.DAGScheduler.$anonfun$runJob$1(DAGScheduler.scala:1245)</SPAN></DIV><DIV><SPAN>at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)</SPAN></DIV><DIV><SPAN>at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:94)</SPAN></DIV><DIV><SPAN>at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:1233)</SPAN></DIV><DIV><SPAN>at org.apache.spark.SparkContext.runJobInternal(SparkContext.scala:2959)</SPAN></DIV><DIV><SPAN>at org.apache.spark.SparkContext.runJob(SparkContext.scala:2942)</SPAN></DIV><DIV><SPAN>at org.apache.spark.SparkContext.runJob(SparkContext.scala:3054)</SPAN></DIV><DIV><SPAN>at org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPythonInternal$4(Dataset.scala:4450)</SPAN></DIV><DIV><SPAN>at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)</SPAN></DIV><DIV><SPAN>at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1709)</SPAN></DIV><DIV><SPAN>at org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPythonInternal$1(Dataset.scala:4454)</SPAN></DIV><DIV><SPAN>at org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPythonInternal$1$adapted(Dataset.scala:4419)</SPAN></DIV><DIV><SPAN>at org.apache.spark.sql.Dataset.$anonfun$withAction$3(Dataset.scala:4544)</SPAN></DIV><DIV><SPAN>at org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:945)</SPAN></DIV><DIV><SPAN>at org.apache.spark.sql.Dataset.$anonfun$withAction$2(Dataset.scala:4542)</SPAN></DIV><DIV><SPAN>at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withCustomExecutionEnv$8(SQLExecution.scala:282)</SPAN></DIV><DIV><SPAN>at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:510)</SPAN></DIV><DIV><SPAN>at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withCustomExecutionEnv$1(SQLExecution.scala:209)</SPAN></DIV><DIV><SPAN>at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:1138)</SPAN></DIV><DIV><SPAN>at org.apache.spark.sql.execution.SQLExecution$.withCustomExecutionEnv(SQLExecution.scala:152)</SPAN></DIV><DIV><SPAN>at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:459)</SPAN></DIV><DIV><SPAN>at org.apache.spark.sql.Dataset.withAction(Dataset.scala:4542)</SPAN></DIV><DIV><SPAN>at org.apache.spark.sql.Dataset.collectAsArrowToPythonInternal(Dataset.scala:4419)</SPAN></DIV><DIV><SPAN>at org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$2(Dataset.scala:4481)</SPAN></DIV><DIV><SPAN>at org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$2$adapted(Dataset.scala:4480)</SPAN></DIV><DIV><SPAN>at org.apache.spark.security.SocketAuthServer$.$anonfun$serveToStream$2(SocketAuthServer.scala:154)</SPAN></DIV><DIV><SPAN>at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)</SPAN></DIV><DIV><SPAN>at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1709)</SPAN></DIV><DIV><SPAN>at org.apache.spark.security.SocketAuthServer$.$anonfun$serveToStream$1(SocketAuthServer.scala:156)</SPAN></DIV><DIV><SPAN>at org.apache.spark.security.SocketAuthServer$.$anonfun$serveToStream$1$adapted(SocketAuthServer.scala:151)</SPAN></DIV><DIV><SPAN>at org.apache.spark.security.SocketFuncServer.handleConnection(SocketAuthServer.scala:125)</SPAN></DIV><DIV><SPAN>at org.apache.spark.security.SocketFuncServer.handleConnection(SocketAuthServer.scala:118)</SPAN></DIV><DIV><SPAN>at org.apache.spark.security.SocketAuthServer$$anon$1.$anonfun$run$4(SocketAuthServer.scala:71)</SPAN></DIV><DIV><SPAN>at scala.util.Try$.apply(Try.scala:213)</SPAN></DIV><DIV><SPAN>at org.apache.spark.security.SocketAuthServer$$anon$1.run(SocketAuthServer.scala:71)</SPAN></DIV><DIV>Caused by: java.io.InvalidClassException: org.apache.spark.sql.sedona_sql.expressions.ST_FlipCoordinates; local class incompatible: stream classdesc serialVersionUID = -8818399722059471005, local class serialVersionUID = -572255679765828529</DIV><DIV><SPAN>at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:699)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2005)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1852)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2186)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readArray(ObjectInputStream.java:2119)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1657)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readArray(ObjectInputStream.java:2119)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1657)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readArray(ObjectInputStream.java:2119)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1657)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)</SPAN></DIV><DIV><SPAN>at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:527)</SPAN></DIV><DIV><SPAN>at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)</SPAN></DIV><DIV><SPAN>at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)</SPAN></DIV><DIV><SPAN>at java.lang.reflect.Method.invoke(Method.java:498)</SPAN></DIV><DIV><SPAN>at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1184)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2322)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503)</SPAN></DIV><DIV><SPAN>at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)</SPAN></DIV><DIV><SPAN>at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:87)</SPAN></DIV><DIV><SPAN>at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:129)</SPAN></DIV><DIV><SPAN>at org.apache.spark.scheduler.ResultTask.$anonfun$runTask$2(ResultTask.scala:72)</SPAN></DIV><DIV><SPAN>at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)</SPAN></DIV><DIV><SPAN>at org.apache.spark.scheduler.ResultTask.$anonfun$runTask$1(ResultTask.scala:72)</SPAN></DIV><DIV><SPAN>at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)</SPAN></DIV><DIV><SPAN>at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)</SPAN></DIV><DIV><SPAN>at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:196)</SPAN></DIV><DIV><SPAN>at org.apache.spark.scheduler.Task.doRunTask(Task.scala:181)</SPAN></DIV><DIV><SPAN>at org.apache.spark.scheduler.Task.$anonfun$run$5(Task.scala:146)</SPAN></DIV><DIV><SPAN>at com.databricks.unity.EmptyHandle$.runWithAndClose(UCSHandle.scala:125)</SPAN></DIV><DIV><SPAN>at org.apache.spark.scheduler.Task.$anonfun$run$1(Task.scala:146)</SPAN></DIV><DIV><SPAN>at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)</SPAN></DIV><DIV><SPAN>at org.apache.spark.scheduler.Task.run(Task.scala:99)</SPAN></DIV><DIV><SPAN>at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$8(Executor.scala:899)</SPAN></DIV><DIV><SPAN>at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1709)</SPAN></DIV><DIV><SPAN>at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:902)</SPAN></DIV><DIV><SPAN>at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)</SPAN></DIV><DIV><SPAN>at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)</SPAN></DIV><DIV><SPAN>at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:797)</SPAN></DIV><DIV><SPAN>at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)</SPAN></DIV><DIV><SPAN>at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)</SPAN></DIV><DIV><SPAN>at java.lang.Thread.run(Thread.java:750)</SPAN></DIV><DIV> </DIV><DIV>  warn(msg)</DIV></DIV></DIV> Mon, 04 Mar 2024 09:06:26 GMT https://community.databricks.com/t5/data-engineering/databricks-error-while-executing-this-line-of-code/m-p/62561#M32000 NT911 2024-03-04T09:06:26Z Re: Databricks Error while executing this line of code https://community.databricks.com/t5/data-engineering/databricks-error-while-executing-this-line-of-code/m-p/62821#M32106 <P>Please suggest correct version of libraries i need to install to avoid above error.</P><P> </P><P> </P> Thu, 07 Mar 2024 08:14:55 GMT https://community.databricks.com/t5/data-engineering/databricks-error-while-executing-this-line-of-code/m-p/62821#M32106 NT911 2024-03-07T08:14:55Z