Upgrading to 11.3lts, Sedona functions throwing null when previously worked fine in 7.3

[Pazuzu7]

I'm in the process of upgrading to 11.3. I'm using spark 3.3.0, scala 2.12, maven and sedona 1.2.0 incubating and followed the installation as outlined by sedona here. Everything was running smoothly in version 7.3 but is currently throwing when reaching sedona functions and providing a null exception at the functions constructor.

To create the clusters, I pass a json config that has the init script location and the confs outlined in the link above. I then point maven to the standard coordinates (from what I understand it should check the jar location in databricks and if they aren't there download from the maven repo).

I have checked the data reaching the function and there are no null values in geometry. I have tried different versions, loading jars directly from dbfs, going through data manually for nulls or unexpected values. I've tried more than one dataset going to this function. All do the same thing.

I apologise if this doesn't make much sense, I don't have a lot of experience with databricks.

[Debayan]

Hi, this looks like an issue with compatibility, could you please check if the dependency packages are also installed?

Also, I saw a note in the end of the link above, could you please confirm if this was done?

Note: You need to install the Sedona libraries via init script because the libraries installed via UI are installed after the cluster has already started, and therefore the classes specified by the config 

spark.sql.extensions

, 

spark.serializer

, and 

spark.kryo.registrator

 are not available at startup time.

[Pazuzu7]

Yes, the Sedina libraries are installed via the init script described in the link. The configs are also set in the same json. I have checked the cluster on creation and the configs are set.

We have quite a few similar things that are going into databricks in the same way with the same library requirements and I see that some of my colleagues have neither used an init script nor set the spark configs and have no issues on running.

[Anonymous]

Hi @William Honeyman​ 

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!