Databricks-Connect Error occurred while running *** java.io.StreamCorruptedException: invalid type code: 00

[DBJmet]

I am using databricks-connect to access a remote cluster. Everything works as expected and I can set breakpoints and interrogate the results, same for when it trys to execute the following code:

val testDF = spark.createDataFrame(spark.sparkContext

.emptyRDD[Row], StructType(

Array(

StructField("prop1", StringType),

StructField("prop2", StringType),

StructField("prop3", StringType),

)))

logLn("Test DF count: " + testDF.count())

I am running this from Intellij. Any help or pointers would be much appreciated.

UPDATE:

The first cluster I was targeting was 11.3.* and I got the error. I tried testing against a 7.3 and it worked as expected. I then tested against 10.4 and got the above error.

Full text of error:

java.io.StreamCorruptedException: invalid type code: 00

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1700)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)

at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:527)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1184)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2322)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)

at java.io.ObjectInputStream.readArray(ObjectInputStream.java:2119)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1657)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)

at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:527)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1184)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2322)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)

at java.io.ObjectInputStream.readArray(ObjectInputStream.java:2119)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1657)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)

at org.apache.spark.sql.util.ProtoSerializer.$anonfun$deserializeObject$1(ProtoSerializer.scala:7248)

...

[Debayan]

Hi, I am not sure if this helps:

https://stackoverflow.com/questions/10775709/java-io-streamcorruptedexception-invalid-type-code-00

https://www.geeksforgeeks.org/how-to-fix-java-io-streamcorruptedexception-invalid-type-code-in-java/

Please tag @Debayan​ with your next response which will notify me. Thank you!

[Anonymous]

Hi @James Metcalf​ 

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future.

Thank you for your participation and let us know if you need any further assistance!