09-02-2022 02:19 PM
Could you please try to update your spark version. For few of the used cases , spark version 2.3.0 was throwing the same error which was fixed in 2.4.0.
Please refer:
https://issues.apache.org/jira/browse/SPARK-24156
