Spark DataFrame apply Databricks geospatial indexing functions

kll — Mon, 15 May 2023 21:13:01 GMT

I have a spark DataFrame with `h3` hex ids and I am trying to obtain the polygon geometries.

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, expr
from pyspark.databricks.sql.functions import *
 
from mosaic import enable_mosaic
enable_mosaic(spark, dbutils)
 
 
# Create a Spark session
spark = SparkSession.builder.appName("Mosaic").getOrCreate()
 
# Create a DataFrame with hex IDs
df = spark.createDataFrame([
    (1, "87422c2a9ffffff"),
    (2, "87422c2a9000000"),
    (3, "87422c2a8ffffff")
], ("id", "h3hex_id"))
 
 
 
sdf2 = sdf1.withColumn("geometry", h3_boundaryaswkt(col("h3hex_id")))
sdf2.sample(fraction=0.1).show()
 
AnalysisException: [H3_NOT_ENABLED] h3_boundaryaswkt is disabled or unsupported. Consider enabling Photon or switch to a tier that supports H3 expressions; 
 
 
sdf2 = sdf1.withColumn("geometry", grid_boundary(col("h3hex_id"), format_name="WKT"))
sdf2.sample(fraction=0.1).show()
 
 
AnalysisException: [UNRESOLVED_COLUMN.WITH_SUGGESTION] A column or function parameter with name `WKT` cannot be resolved. Did you mean one of the following? ..

I have installed `databricks-mosaic 0.3.10` on the cluster.

How do I resolve the exception and apply the function spark DataFrame?

https://databrickslabs.github.io/mosaic/api/spatial-indexing.html

https://docs.databricks.com/sql/language-manual/functions/h3_boundaryaswkt.html#examples

topic Spark DataFrame apply Databricks geospatial indexing functions in Data Engineering

Spark DataFrame apply Databricks geospatial indexing functions