topic Re: Create a simple Geospatial Table with Geography type column in Data Engineering

Create a simple Geospatial Table with Geography type column

lakshgisprog — Tue, 05 Nov 2024 17:30:54 GMT

Hello All

I am looking for guidance on creating an simple US states table with shape as Geography column type. I do not want to use Apache Sedona (due to cluster limitations). I am going to create an Node JS application which is going to query this geospatial data.

Re: Create a simple Geospatial Table with Geography type column

filipniziol — Tue, 05 Nov 2024 18:23:52 GMT

Hi @lakshgisprog ,

Recently there was a technical article on geospatial data processing. You may start with that one:
https://community.databricks.com/t5/technical-blog/from-raw-to-refined-processing-overture-maps-geospatial-data-on/ba-p/90838

Re: Create a simple Geospatial Table with Geography type column

lakshgisprog — Tue, 05 Nov 2024 19:04:11 GMT

Thank you for prompt response. Yes, I have gone through the blog. I followed the same process ; for examples, buildings table have geometry which stores as binary type. My question, how to store geometry in 'Geography' type. The GEOGRAPHY data type follows the WGS 84 standard (spatial reference ID 4326; for details, see https://epsg.io/4326).

My use case is creating an Node JS application to query the geospatial data and perform spatial operations.

Second part of Danny Blog is using Sedona for dynamic segmentation. Since my compute is shared, I cannot use external library rather want to explore native spatial types and operation.

Let me know if you have some pointers.

Re: Create a simple Geospatial Table with Geography type column

filipniziol — Wed, 06 Nov 2024 08:30:41 GMT

Hi @lakshgisprog ,

In databricks there is no GEOGRAPHY data type.
Please check the documentation for the available data types:
https://docs.databricks.com/en/sql/language-manual/sql-ref-datatypes.html

What you can do is:
1. You can store GEOGRAPHY as BINARY or STRING in delta table

2. After reading the data from the table you will convert it in your Node JS application into GEOGRAPHY data type

Also, for transformations on Databricks side you should be able to use ST expressions and H3 library on the shared cluster with photon-enabled clusters, as H3 library is installed by default:
https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/sql-ref-h3-geospatial-functions