Hello:)
we are trying to run an existing working flow that works currently on EMR, on databricks.
we use LTS 10.4, and when loading the data we get the following error:
at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:365)
Caused by: java.lang.RuntimeException: org.apache.spark.sql.catalyst.util.GenericArrayData is not a valid external type for schema of array<struct<id:string,name:string,name_type:string>>
at com.databricks.sql.expressions.codegen.RuntimeUtils.assertTrue(RuntimeUtils.java:77)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.MapObjects_0$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$Serializer.apply(ExpressionEncoder.scala:233)
... 19 more
it's important to note that the same code works on emr with spark version 3.2.1. we tried looking at the data, removing some of the columns - but the data is consistent with the structure.
did anyone encounter this and could provide assistance in identifying the cause? we'd be happy to share more of the stack if relevant.
Thanks in advance!
