GEOMETRY column type breaks all access to table from Spark

[sfishel18]

Hello, 

I have a Databricks table with a column using the new GEOMETRY type. When I try to access this table from a Spark workload, I am not able to describe the table or operate on any of its columns. My Spark config is the following, per the Databricks docs:

spark.sql.catalog.spark_catalog org.apache.spark.sql.delta.catalog.DeltaCatalog
spark.sql.catalog.my_catalog io.unitycatalog.spark.UCSingleCatalog
spark.sql.catalog.my_catalog.uri <Databricks workspace URL>/api/2.1/unity-catalog
spark.sql.catalog.my_catalog.token <Databricks PAT>

If I try to access the table from my Spark workload - even to examine its schema or operate on non-geometry columns - I get an error like this:

io.unitycatalog.client.ApiException: com.fasterxml.jackson.databind.exc.ValueInstantiationException: Cannot construct instance of `io.unitycatalog.client.model.ColumnTypeName`, problem: Unexpected value 'GEOMETRY'
 at [Source: (jdk.internal.net.http.ResponseSubscribers$HttpResponseInputStream); line: 1, column: 8354] (through reference chain: io.unitycatalog.client.model.TableInfo["columns"]->java.util.ArrayList[13]->io.unitycatalog.client.model.ColumnInfo["type_name"])
	at io.unitycatalog.client.api.TablesApi.getTableWithHttpInfo(TablesApi.java:273)
	at io.unitycatalog.client.api.TablesApi.getTable(TablesApi.java:241)
	at io.unitycatalog.spark.UCProxy.loadTable(UCSingleCatalog.scala:237)
	at org.apache.spark.sql.connector.catalog.DelegatingCatalogExtension.loadTable(DelegatingCatalogExtension.java:73)
	at org.apache.spark.sql.delta.catalog.DeltaCatalog.super$loadTable(DeltaCatalog.scala:229)
	at org.apache.spark.sql.delta.catalog.DeltaCatalog.$anonfun$loadTable$1(DeltaCatalog.scala:229)
	at org.apache.spark.sql.delta.metering.DeltaLogging.recordFrameProfile(DeltaLogging.scala:171)
	at org.apache.spark.sql.delta.metering.DeltaLogging.recordFrameProfile$(DeltaLogging.scala:169)
	at org.apache.spark.sql.delta.catalog.DeltaCatalog.recordFrameProfile(DeltaCatalog.scala:67)
	at org.apache.spark.sql.delta.catalog.DeltaCatalog.loadTable(DeltaCatalog.scala:228)
	at io.unitycatalog.spark.UCSingleCatalog.loadTable(UCSingleCatalog.scala:73)
	at org.apache.spark.sql.connector.catalog.CatalogV2Util$.getTable(CatalogV2Util.scala:363)
	at org.apache.spark.sql.connector.catalog.CatalogV2Util$.loadTable(CatalogV2Util.scala:337)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.$anonfun$lookupTableOrView$2(Analyzer.scala:1228)
[...]

[sfishel18]

I have filed an issue here: https://github.com/unitycatalog/unitycatalog/issues/1077. But also wanted to ask for help here since this is a Databricks-specific column type.

[NandiniN]

Hi @sfishel18,

This error surely is should be handled better. However the fix will have to go via UC oss (for any unknown type)

Meanwhile, where are you trying to access the table? Where is your Spark workload running?

Thanks!

[sfishel18]

Hi @NandiniN,

My workload is running on the Wherobots platform.

Simon

[NandiniN]

Reading the UC OSS doc - https://docs.unitycatalog.io/usage/tables/formats/

it does not support all data types

columns

The columns of the table in SQL-like format "column_name column_data_type". Supported data types include BOOLEAN, BYTE, SHORT, INT, LONG, FLOAT, DOUBLE, DATE, TIMESTAMP, TIMESTAMP_NTZ, STRING, BINARY, DECIMAL. Separate multiple columns with a comma (e.g., "id INT, name STRING")

Databricks sdk doc https://databricks-sdk-py.readthedocs.io/en/latest/dbdataclasses/catalog.html#databricks.sdk.service... does support it.

https://docs.databricks.com/aws/en/sql/language-manual/data-types/geometry-type is supported on Databricks Runtime 17.1 and above.

And can you please share the details of the table as well, Iceberg tables do not support GEOMETRY columns.

[sfishel18]

Hi @NandiniN,

To my understanding, there is no way to use the Databricks SDK as a catalog implementation in an external Spark workload, correct?

The table I'm using is a Delta table I created from a Databricks notebook. The rows represent some areas-of-interest for spatial analysis, with my geometry-typed column encoding the polygon of the area.