Hello,

I'm using, in my IDE, Databricks Connect version 9.1LTS ML to connect to a databricks cluster with spark version 3.1 and download a spark model that's been trained and saved using mlflow.

So it seems like it's able to find a copy the model, but then it's wrong. The same works in a databricks notebook are goods, the problem only occurs using databricks connect in my IDE.

We are getting the same error in different repositories with different models. It started to appear recently.

I have the same problem in other environment with cluster 10.4LTS ML and databricks-connect 10.4.6.

Do you have an idea ?

code :

mlflow.set_tracking_uri("databricks")

model_path = 'dbfs:/databricks/mlflow-tracking/197830957424395/7c5e692873874dadae4f67f44c1aa310/artifacts/rfModel'

model_res = mlflow.spark.load_model(model_path)

See the StackTraceError :

2022/10/06 15:17:11 INFO mlflow.spark: File 'dbfs:/databricks/mlflow-tracking/197830957424395/7c5e692873874dadae4f67f44c1aa310/artifacts/rfModel/sparkml' not found on DFS. Will attempt to upload the file.

22/10/06 15:17:39 WARN DBFS: DBFS create on /tmp/mlflow/f020cb9a-47b2-49ee-8b12-cf2754db61a9/metadata/part-00000 took 2299 ms

22/10/06 15:17:42 WARN DBFS: DBFS create on /tmp/mlflow/f020cb9a-47b2-49ee-8b12-cf2754db61a9/metadata/_SUCCESS took 1687 ms

22/10/06 15:17:46 WARN DBFS: DBFS mkdirs on /tmp/mlflow/f020cb9a-47b2-49ee-8b12-cf2754db61a9/stages/0_RandomForestClassifier_77e9017cbf4d took 2302 ms

2022/10/06 15:19:13 INFO mlflow.spark: Copied SparkML model to /tmp/mlflow/f020cb9a-47b2-49ee-8b12-cf2754db61a9

View job details at ........https....

View job details at ........ https .....

22/10/06 15:19:16 ERROR Instrumentation: java.io.StreamCorruptedException: invalid type code: 00

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1698)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)

at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:488)

at sun.reflect.GeneratedMethodAccessor419.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1184)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2296)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)

at java.io.ObjectInputStream.readArray(ObjectInputStream.java:2093)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1655)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)

at java.io.ObjectInputStream.readArray(ObjectInputStream.java:2093)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1655)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)

at org.apache.spark.sql.util.ProtoSerializer.$anonfun$deserializeObject$1(ProtoSerializer.scala:6631)

at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)

at org.apache.spark.sql.util.ProtoSerializer.deserializeObject(ProtoSerializer.scala:6616)

at com.databricks.service.SparkServiceRPCHandler.execute0(SparkServiceRPCHandler.scala:728)

at com.databricks.service.SparkServiceRPCHandler.$anonfun$executeRPC0$1(SparkServiceRPCHandler.scala:477)

at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)

at com.databricks.service.SparkServiceRPCHandler.executeRPC0(SparkServiceRPCHandler.scala:372)

at com.databricks.service.SparkServiceRPCHandler$$anon$2.call(SparkServiceRPCHandler.scala:323)

at com.databricks.service.SparkServiceRPCHandler$$anon$2.call(SparkServiceRPCHandler.scala:309)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at com.databricks.service.SparkServiceRPCHandler.$anonfun$executeRPC$1(SparkServiceRPCHandler.scala:359)

at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)

at com.databricks.service.SparkServiceRPCHandler.executeRPC(SparkServiceRPCHandler.scala:336)

at com.databricks.service.SparkServiceRPCServlet.doPost(SparkServiceRPCServer.scala:167)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:190)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)

at org.eclipse.jetty.server.Server.handle(Server.java:516)

at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)

at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)

at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)

at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)

at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)

at java.lang.Thread.run(Thread.java:748)

...

py4j.protocol.Py4JJavaError: An error occurred while calling o588.load.

: java.io.StreamCorruptedException: invalid type code: 00

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1698)

Thanks for your help.