topic Re: Upgrading to 11.3lts, Sedona functions throwing null when previously worked fine in 7.3 in Data Engineering

Upgrading to 11.3lts, Sedona functions throwing null when previously worked fine in 7.3

Pazuzu7 — Fri, 10 Feb 2023 00:57:15 GMT

I'm in the process of upgrading to 11.3. I'm using spark 3.3.0, scala 2.12, maven and sedona 1.2.0 incubating and followed the installation as outlined by sedona here. Everything was running smoothly in version 7.3 but is currently throwing when reaching sedona functions and providing a null exception at the functions constructor.

To create the clusters, I pass a json config that has the init script location and the confs outlined in the link above. I then point maven to the standard coordinates (from what I understand it should check the jar location in databricks and if they aren't there download from the maven repo).

I have checked the data reaching the function and there are no null values in geometry. I have tried different versions, loading jars directly from dbfs, going through data manually for nulls or unexpected values. I've tried more than one dataset going to this function. All do the same thing.

I apologise if this doesn't make much sense, I don't have a lot of experience with databricks.

Re: Upgrading to 11.3lts, Sedona functions throwing null when previously worked fine in 7.3

Pazuzu7 — Mon, 13 Feb 2023 09:08:44 GMT

Yes, the Sedina libraries are installed via the init script described in the link. The configs are also set in the same json. I have checked the cluster on creation and the configs are set.

We have quite a few similar things that are going into databricks in the same way with the same library requirements and I see that some of my colleagues have neither used an init script nor set the spark configs and have no issues on running.

Re: Upgrading to 11.3lts, Sedona functions throwing null when previously worked fine in 7.3

Anonymous — Mon, 10 Apr 2023 08:43:27 GMT

Hi @William Honeyman

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!

Re: Upgrading to 11.3lts, Sedona functions throwing null when previously worked fine in 7.3

Debayan — Mon, 13 Feb 2023 05:04:25 GMT

Hi, this looks like an issue with compatibility, could you please check if the dependency packages are also installed?

Also, I saw a note in the end of the link above, could you please confirm if this was done?

Note: You need to install the Sedona libraries via init script because the libraries installed via UI are installed after the cluster has already started, and therefore the classes specified by the config

spark.sql.extensions

spark.serializer

, and

spark.kryo.registrator

are not available at startup time.