DBR 17.1 Spatial SQL Functions and Apache Sedona

[der]

I noticed in the DBR 17.1 release notes that ST geospatial functions are now in public preview - great news for us since this means native support in Databricks.

https://docs.databricks.com/aws/en/release-notes/runtime/17.1#expanded-spatial-sql-expressions-and-g...

We're currently using Apache Sedona 1.7.2 with DBR 16.4, and I have a few questions:

1. Can Apache Sedona (already Spark 4.0-ready: https://github.com/apache/sedona/tree/master/spark/spark-4.0 ) be used alongside DBR 17.1?
2. When both Sedona and DBR 17.1+ are active, which SQL functions are executed?
3. Will additional ST functions be introduced, and is there a plan for raster functions?
   https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-st-geospatial-functions-alpha

[der]

@mjohns @dbkent do you know more about Apache Sedona and DBR SQL Spatial Functions?

[mjohns]

Here are a few answers, feel free to hit me up on LinkedIn (michaeljohns2) if you want to discuss more particulars wrt Databricks geospatial.

1. Looks like Sedona 1.8.0 is the release to watch for with Spark 4.0 support, see https://github.com/apache/sedona/milestones (2 outstanding tasks as of this post).
2. Sedona has the ability to "step-on" built-in functions in Databricks classic compute when registering SQL expressions of the same name as those provided by product. Two options: (a) test the 'Class' provider of a given function using `describe function extended ST_<func>` and (b) consider using CARTO's Analytics Toolbox which rewraps and prefixes the registered SQL expressions as `sedona_*` (AT is a mix of free and paid capabilities). Perhaps Sedona will introduce built-in function prefixing option in a future release?
3. Yes, additional ST_ functions have been identified and are planned. Right now, product is most focused on spatial performance as we advance the feature from Public Preview to GA. Raster is something product is gathering requirements on but no commitments on productizing as of yet. For my role as SME Leader, we are WIP for this quarter on a revamp of older Mosaic raster capabilities that accomplishes the following: (a) moves to DBR17.3 LTS support, (b) works with Volumes, and (c) uses GDAL 3.11.3 ... among other improvements. Stay tuned on that!

Michael Johns | Geospatial Specialist Leader