topic Re: GEOMETRY column type breaks all access to table from Spark in Data Engineering

GEOMETRY column type breaks all access to table from Spark

sfishel18 — Fri, 12 Sep 2025 18:43:38 GMT

Hello,

I have a Databricks table with a column using the new GEOMETRY type. When I try to access this table from a Spark workload, I am not able to describe the table or operate on any of its columns. My Spark config is the following, per the Databricks docs:

spark.sql.catalog.spark_catalog org.apache.spark.sql.delta.catalog.DeltaCatalog
spark.sql.catalog.my_catalog io.unitycatalog.spark.UCSingleCatalog
spark.sql.catalog.my_catalog.uri <Databricks workspace URL>/api/2.1/unity-catalog
spark.sql.catalog.my_catalog.token <Databricks PAT>

If I try to access the table from my Spark workload - even to examine its schema or operate on non-geometry columns - I get an error like this:

io.unitycatalog.client.ApiException: com.fasterxml.jackson.databind.exc.ValueInstantiationException: Cannot construct instance of `io.unitycatalog.client.model.ColumnTypeName`, problem: Unexpected value 'GEOMETRY'
 at [Source: (jdk.internal.net.http.ResponseSubscribers$HttpResponseInputStream); line: 1, column: 8354] (through reference chain: io.unitycatalog.client.model.TableInfo["columns"]->java.util.ArrayList[13]->io.unitycatalog.client.model.ColumnInfo["type_name"])
	at io.unitycatalog.client.api.TablesApi.getTableWithHttpInfo(TablesApi.java:273)
	at io.unitycatalog.client.api.TablesApi.getTable(TablesApi.java:241)
	at io.unitycatalog.spark.UCProxy.loadTable(UCSingleCatalog.scala:237)
	at org.apache.spark.sql.connector.catalog.DelegatingCatalogExtension.loadTable(DelegatingCatalogExtension.java:73)
	at org.apache.spark.sql.delta.catalog.DeltaCatalog.super$loadTable(DeltaCatalog.scala:229)
	at org.apache.spark.sql.delta.catalog.DeltaCatalog.$anonfun$loadTable$1(DeltaCatalog.scala:229)
	at org.apache.spark.sql.delta.metering.DeltaLogging.recordFrameProfile(DeltaLogging.scala:171)
	at org.apache.spark.sql.delta.metering.DeltaLogging.recordFrameProfile$(DeltaLogging.scala:169)
	at org.apache.spark.sql.delta.catalog.DeltaCatalog.recordFrameProfile(DeltaCatalog.scala:67)
	at org.apache.spark.sql.delta.catalog.DeltaCatalog.loadTable(DeltaCatalog.scala:228)
	at io.unitycatalog.spark.UCSingleCatalog.loadTable(UCSingleCatalog.scala:73)
	at org.apache.spark.sql.connector.catalog.CatalogV2Util$.getTable(CatalogV2Util.scala:363)
	at org.apache.spark.sql.connector.catalog.CatalogV2Util$.loadTable(CatalogV2Util.scala:337)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.$anonfun$lookupTableOrView$2(Analyzer.scala:1228)
[...]

Re: GEOMETRY column type breaks all access to table from Spark

sfishel18 — Fri, 12 Sep 2025 18:44:00 GMT

I have filed an issue here: https://github.com/unitycatalog/unitycatalog/issues/1077. But also wanted to ask for help here since this is a Databricks-specific column type.

Re: GEOMETRY column type breaks all access to table from Spark

NandiniN — Fri, 19 Sep 2025 18:56:26 GMT

Hi @sfishel18,

This error surely is should be handled better. However the fix will have to go via UC oss (for any unknown type)

Meanwhile, where are you trying to access the table? Where is your Spark workload running?

Thanks!

Re: GEOMETRY column type breaks all access to table from Spark

sfishel18 — Fri, 19 Sep 2025 19:16:34 GMT

Hi @NandiniN,

My workload is running on the Wherobots platform.

Simon

Re: GEOMETRY column type breaks all access to table from Spark

NandiniN — Fri, 19 Sep 2025 19:32:04 GMT

Reading the UC OSS doc - https://docs.unitycatalog.io/usage/tables/formats/

it does not support all data types

columns The columns of the table in SQL-like format "column_name column_data_type". Supported data types include BOOLEAN, BYTE, SHORT, INT, LONG, FLOAT, DOUBLE, DATE, TIMESTAMP, TIMESTAMP_NTZ, STRING, BINARY, DECIMAL. Separate multiple columns with a comma (e.g., "id INT, name STRING")

Databricks sdk doc https://databricks-sdk-py.readthedocs.io/en/latest/dbdataclasses/catalog.html#databricks.sdk.service.catalog.ColumnTypeName does support it.

https://docs.databricks.com/aws/en/sql/language-manual/data-types/geometry-type is supported on Databricks Runtime 17.1 and above.

And can you please share the details of the table as well, Iceberg tables do not support GEOMETRY columns.

Re: GEOMETRY column type breaks all access to table from Spark

sfishel18 — Mon, 22 Sep 2025 16:51:05 GMT

Hi @NandiniN,

To my understanding, there is no way to use the Databricks SDK as a catalog implementation in an external Spark workload, correct?

The table I'm using is a Delta table I created from a Databricks notebook. The rows represent some areas-of-interest for spatial analysis, with my geometry-typed column encoding the polygon of the area.