Hubert-Dudek
Databricks MVP

Additionally, databricks cluster need that config:

spark.driver.extraJavaOptions -Dio.netty.tryReflectionSetAccessible=true

spark.executor.extraJavaOptions -Dio.netty.tryReflectionSetAccessible=true


My blog: https://databrickster.medium.com/