topic Re: DBR 17.1 Spatial SQL Functions and Apache Sedona in Data Engineering

DBR 17.1 Spatial SQL Functions and Apache Sedona

der — Tue, 19 Aug 2025 12:39:29 GMT

I noticed in the DBR 17.1 release notes that ST geospatial functions are now in public preview - great news for us since this means native support in Databricks.

https://docs.databricks.com/aws/en/release-notes/runtime/17.1#expanded-spatial-sql-expressions-and-geometry-and-geography-data-types

We're currently using Apache Sedona 1.7.2 with DBR 16.4, and I have a few questions:

Can Apache Sedona (already Spark 4.0-ready: https://github.com/apache/sedona/tree/master/spark/spark-4.0 ) be used alongside DBR 17.1?
When both Sedona and DBR 17.1+ are active, which SQL functions are executed?
Will additional ST functions be introduced, and is there a plan for raster functions?
https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-st-geospatial-functions-alpha

Re: DBR 17.1 Spatial SQL Functions and Apache Sedona

der — Tue, 02 Sep 2025 06:48:05 GMT

@mjohns @dbkent do you know more about Apache Sedona and DBR SQL Spatial Functions?

Re: DBR 17.1 Spatial SQL Functions and Apache Sedona

mjohns — Tue, 02 Sep 2025 19:54:56 GMT

Here are a few answers, feel free to hit me up on LinkedIn (michaeljohns2) if you want to discuss more particulars wrt Databricks geospatial.

Looks like Sedona 1.8.0 is the release to watch for with Spark 4.0 support, see https://github.com/apache/sedona/milestones (2 outstanding tasks as of this post).
Sedona has the ability to "step-on" built-in functions in Databricks classic compute when registering SQL expressions of the same name as those provided by product. Two options: (a) test the 'Class' provider of a given function using `describe function extended ST_<func>` and (b) consider using CARTO's Analytics Toolbox which rewraps and prefixes the registered SQL expressions as `sedona_*` (AT is a mix of free and paid capabilities). Perhaps Sedona will introduce built-in function prefixing option in a future release?
Yes, additional ST_ functions have been identified and are planned. Right now, product is most focused on spatial performance as we advance the feature from Public Preview to GA. Raster is something product is gathering requirements on but no commitments on productizing as of yet. For my role as SME Leader, we are WIP for this quarter on a revamp of older Mosaic raster capabilities that accomplishes the following: (a) moves to DBR17.3 LTS support, (b) works with Volumes, and (c) uses GDAL 3.11.3 ... among other improvements. Stay tuned on that!

Michael Johns | Geospatial Specialist Leader