I'm facing an issue when I want to show a dataframe with JSON content.
All this happens when the script runs in databricks-connect from VS Code.
Basically, I would like any help or guidance to get this run as it should be.
Thanks in advance.
This is how the cluster is configured.
Cluster Azure Databricks runtime 10.4
Workers 2-8 Standard_DS3_v2 14GB Memory, 4 cores
Driver Standard_DS3_v2 14GB Memory, 4 cores
Spark config.
spark.databricks.service.server.enabled true
spark.databricks.service.port 8787
spark.hadoop.datanucleus.connectionPoolingType hikari
spark.databricks.delta.preview.enabled true
On my local computer
I installed databricks-connect using pip install databricks-connect 10.4.*
It is configured as per documentation indicates. Azure databricks-connect setup
When I run databricks-connect test passes without any failure
The code I'm trying to run is this:
from pyspark.sql import SparkSession
from pyspark.sql.types import *
from datetime import date
nested_row = ['{"name":"Yin","address":{"city":"Columbus","state":"Ohio"}}']
nested_struct = StructType([
StructField("address",StructType([
StructField("city",StringType(),True),
StructField("state",StringType(),True)
]),True),
StructField("name",StringType(),True)
])
nested_rdd = sc.parallelize(nested_row)
df_json = spark.read.json(nested_rdd,nested_struct)
df_json.printSchema()
df_json.show()
everything runs fine until the printSchema, then when I want to show the dataframe throws an exception.
Traceback (most recent call last):
File "c:\Data\projects\vcode\gbrx-dbconnect\dbc1.py", line 76, in <module>
df_json.show()
File "c:\Data\projects\vcode\gbrx-dbconnect\venv\lib\site-packages\pyspark\sql\dataframe.py", line 502, in show
print(self._jdf.showString(n, 20, vertical))
File "c:\Data\projects\vcode\gbrx-dbconnect\venv\lib\site-packages\py4j\java_gateway.py", line 1304, in __call__
return_value = get_return_value(
File "c:\Data\projects\vcode\gbrx-dbconnect\venv\lib\site-packages\pyspark\sql\utils.py", line 117, in deco
return f(*a, **kw)
File "c:\Data\projects\vcode\gbrx-dbconnect\venv\lib\site-packages\py4j\protocol.py", line 326, in get_return_value
raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling o46.showString.
: java.lang.ClassCastException: cannot assign instance of java.lang.String to field org.apache.spark.sql.catalyst.json.JSONOptions.lineSeparatorInRead of type scala.Option in instance of org.apache.spark.sql.catalyst.json.JSONOptions
at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2301)
at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1431)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2411)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
at java.io.ObjectInputStream.readArray(ObjectInputStream.java:2093)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1655)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
at java.io.ObjectInputStream.readArray(ObjectInputStream.java:2093)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1655)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)
at org.apache.spark.sql.util.ProtoSerializer.$anonfun$deserializeObject$1(ProtoSerializer.scala:7055)
