topic Re: Facing issues while upgrading DBR version from 9.1 LTS to 15.4 LTS in Get Started Discussions

Facing issues while upgrading DBR version from 9.1 LTS to 15.4 LTS

Kuchnhi — Tue, 08 Apr 2025 08:40:05 GMT

Dear all,

I am upgrading DBR version from 9.1 LTS to 15.4 LTS in Azure Databricks. for that I have created a new cluster with 15.4 DBR attached init script for installing application dependencies. Cluster has started successfully but it takes 30 min. So, when I run my application I am getting below error. what is the reason of issue and how to resolve that? Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.runJob.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3) (10.141.5.11 executor 5): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
File "/databricks/spark/python/pyspark/serializers.py", line 192, in _read_with_length
return self.loads(obj)
^^^^^^^^^^^^^^^
File "/databricks/spark/python/pyspark/serializers.py", line 572, in loads
return cloudpickle.loads(obj, encoding=encoding)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'dbruntime'

Re: Facing issues while upgrading DBR version from 9.1 LTS to 15.4 LTS

Louis_Frolio — Tue, 08 Apr 2025 18:25:08 GMT

I suggest troubleshooting the application dependencies and conflicts with libraries installed on 15.4. Seems to me there is a conflict.

Re: Facing issues while upgrading DBR version from 9.1 LTS to 15.4 LTS

Kuchnhi — Wed, 09 Apr 2025 06:22:51 GMT

Hi @Louis_Frolio Thanks for replying.

I tried by removing all predefined libraries from init script and after starting the cluster, installed required libraries manually. but the issue is still same. I am sharing complete error details below. could you all please look in detail and share some more insights. your Response will be much helpful to us.

Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.runJob.

: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 7) (10.141.5.10 executor 0): org.apache.spark.api.python.PythonException: Traceback (most recent call last):

File "/databricks/spark/python/pyspark/serializers.py", line 192, in _read_with_length

return self.loads(obj)

^^^^^^^^^^^^^^^

File "/databricks/spark/python/pyspark/serializers.py", line 572, in loads

return cloudpickle.loads(obj, encoding=encoding)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

ModuleNotFoundError: No module named 'dbruntime'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/databricks/spark/python/pyspark/worker.py", line 1980, in main

process()

File "/databricks/spark/python/pyspark/worker.py", line 1972, in process

serializer.dump_stream(out_iter, outfile)

File "/databricks/spark/python/pyspark/serializers.py", line 356, in dump_stream

vs = list(itertools.islice(iterator, batch))

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/databricks/spark/python/pyspark/core/rdd.py", line 2757, in takeUpToNumLeft

yield next(iterator)

^^^^^^^^^^^^^^

File "/databricks/spark/python/pyspark/serializers.py", line 162, in load_stream

yield self._read_with_length(stream)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/databricks/spark/python/pyspark/serializers.py", line 196, in _read_with_length

raise SerializationError("Caused by " + traceback.format_exc())

pyspark.serializers.SerializationError: Caused by Traceback (most recent call last):

File "/databricks/spark/python/pyspark/serializers.py", line 192, in _read_with_length

return self.loads(obj)

^^^^^^^^^^^^^^^

File "/databricks/spark/python/pyspark/serializers.py", line 572, in loads

return cloudpickle.loads(obj, encoding=encoding)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

ModuleNotFoundError: No module named 'dbruntime'

at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:560)

at org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:968)

at org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:953)

at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:516)

at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)

at scala.collection.Iterator.foreach(Iterator.scala:943)

at scala.collection.Iterator.foreach$(Iterator.scala:943)

at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)

at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)

at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)

at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105)

at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49)

at scala.collection.TraversableOnce.to(TraversableOnce.scala:366)

at scala.collection.TraversableOnce.to$(TraversableOnce.scala:364)

at org.apache.spark.InterruptibleIterator.to(InterruptibleIterator.scala:28)

at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:358)

at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:358)

at org.apache.spark.InterruptibleIterator.toBuffer(InterruptibleIterator.scala:28)

at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:345)

at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:339)

at org.apache.spark.InterruptibleIterator.toArray(InterruptibleIterator.scala:28)

at org.apache.spark.api.python.PythonRDD$.$anonfun$collectPartitions$1(PythonRDD.scala:234)

at org.apache.spark.SparkContext.$anonfun$runJob$2(SparkContext.scala:3208)

at org.apache.spark.scheduler.ResultTask.$anonfun$runTask$3(ResultTask.scala:82)

at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)

at org.apache.spark.scheduler.ResultTask.$anonfun$runTask$1(ResultTask.scala:82)

at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)

at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)

at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:225)

at org.apache.spark.scheduler.Task.doRunTask(Task.scala:199)

at org.apache.spark.scheduler.Task.$anonfun$run$5(Task.scala:161)

at com.databricks.unity.EmptyHandle$.runWithAndClose(UCSHandle.scala:134)

at org.apache.spark.scheduler.Task.$anonfun$run$1(Task.scala:155)

at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)

at org.apache.spark.scheduler.Task.run(Task.scala:102)

at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$10(Executor.scala:1043)

at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)

at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)

at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:111)

at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:1046)

at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)

at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)

at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:933)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:750)

Driver stacktrace:

at org.apache.spark.scheduler.DAGScheduler.$anonfun$failJobAndIndependentStages$1(DAGScheduler.scala:4043)

at scala.Option.getOrElse(Option.scala:189)

at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:4041)

at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:3954)

at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:3941)

at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)

at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)

at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)

at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:3941)

at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1767)

at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1750)

at scala.Option.foreach(Option.scala:407)

at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1750)

at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:4302)

at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:4204)

at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:4190)

at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:55)

at org.apache.spark.scheduler.DAGScheduler.$anonfun$runJob$1(DAGScheduler.scala:1413)

at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)

at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:94)

at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:1401)

at org.apache.spark.SparkContext.runJobInternal(SparkContext.scala:3168)

at org.apache.spark.SparkContext.runJob(SparkContext.scala:3149)

at org.apache.spark.SparkContext.runJob(SparkContext.scala:3189)

at org.apache.spark.SparkContext.runJob(SparkContext.scala:3208)

at org.apache.spark.api.python.PythonRDD$.collectPartitions(PythonRDD.scala:234)

at org.apache.spark.api.python.PythonRDD$.runJob(PythonRDD.scala:254)

at org.apache.spark.api.python.PythonRDD.runJob(PythonRDD.scala)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)

at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:397)

at py4j.Gateway.invoke(Gateway.java:306)

at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)

at py4j.commands.CallCommand.execute(CallCommand.java:79)

at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:199)

at py4j.ClientServerConnection.run(ClientServerConnection.java:119)

at java.lang.Thread.run(Thread.java:750)

Caused by: org.apache.spark.api.python.PythonException: Traceback (most recent call last):