sedona/shapely error Unknown WKB type 16

[amitca71]

Hi,

i stream data from postgis to s3 using debezium. postgis->debezium->s3->spark(databricks)

once read it i decode it and i can see that the binary representation is similiar to what i have in postgis, on a wkb formated column.

once i try to read it either using sedona or shapely, i get same error (Unknown WKB type 16 ).

the following is how i read the data from s3

this is how i read the wkb column:

select distinct hex(unbase64(after.geom.wkb)) as geom from ....

the result is:

geom ='01100000A0AF0E0000140000000111000080010000000400000035F609BBF4691F4137C9F7AD900FCD4000000060DFDEFA3F08CDDF0AFB691F41BA09018EF90FCD4000000060DFDEFA3F08CDDF0AFB691F41BA09018EF90FCD4072BC746BD03B104035F609BBF4691F4137C9F7AD900FCD4000000060DFDEFA3F0111000080010000000400000035F609BBF4691F4137C9F7AD900FCD4000000060DFDEFA3F08CDDF0AFB691F41BA09018EF90FCD4072BC746BD03B104035F609BBF4691F4137C9F7AD900FCD4072BC746BD03B104035F609BBF4691F4137C9F7AD900FCD4000000060DFDEFA3F01110000800100000004000000005CC17ADD691F4100A0C09E2915CD4000000060DFDEFA3F35F609BBF4691F4137C9F7AD900FCD4000000060DFDEFA3F35F609BBF4691F4137C9F7AD900FCD4072BC746BD03B1040005CC17ADD691F4100A0C09E2915CD4000000060DFDEFA3F01110000800100000004000000005CC17ADD691F4100A0C09E2915CD4000000060DFDEFA3F35F609BBF4691F4137C9F7AD900FCD4072BC746BD03B1040005CC17ADD691F4100A0C09E2915CD4072BC746BD03B1040005CC17ADD691F4100A0C09E2915CD4000000060DFDEFA3F01110000800100000004000000EF68F0F8E3691F410436E5809515CD4000000060DFDEFA3F005CC17ADD691F4100A0C09E2915CD4000000060DFDEFA3F005CC17ADD691F4100A0C09E2915CD4072BC746BD03B1040EF68F0F8E3691F410436E5809515CD4000000060DFDEFA3F01110000800100000004000000EF68F0F8E3691F410436E5809515CD4000000060DFDEFA3F005CC17ADD691F4100A0C09E2915CD4072BC746BD03B1040EF68F0F8E3691F410436E5809515CD4072BC746BD03B1040EF68F0F8E3691F410436E5809515CD4000000060DFDEFA3F01110000800100000004000000EFB06D03FB691F4104F663830910CD4000000060DFDEFA3FEF68F0F8E3691F410436E5809515CD4000000060DFDEFA3FEF68F0F8E3691F410436E5809515CD4072BC746BD03B1040EFB06D03FB691F4104F663830910CD4000000060DFDEFA3F01110000800100000004000000EFB06D03FB691F4104F663830910CD4000000060DFDEFA3FEF68F0F8E3691F410436E5809515CD4072BC746BD03B1040EFB06D03FB691F4104F663830910CD4072BC746BD03B1040EFB06D03FB691F4104F663830910CD4000000060DFDEFA3F01110000800100000004000000D37A14D5FA691F4183A048810610CD4000000060DFDEFA3FEFB06D03FB691F4104F663830910CD4000000060DFDEFA3FEFB06D03FB691F4104F663830910CD4072BC746BD03B1040D37A14D5FA691F4183A048810610CD4000000060DFDEFA3F01110000800100000004000000D37A14D5FA691F4183A048810610CD4000000060DFDEFA3FEFB06D03FB691F4104F663830910CD4072BC746BD03B1040D37A14D5FA691F4183A048810610CD4072BC746BD03B1040D37A14D5FA691F4183A048810610CD4000000060DFDEFA3F0111000080010000000400000008CDDF0AFB691F41BA09018EF90FCD4000000060DFDEFA3FD37A14D5FA691F4183A048810610CD4000000060DFDEFA3FD37A14D5FA691F4183A048810610CD4072BC746BD03B104008CDDF0AFB691F41BA09018EF90FCD4000000060DFDEFA3F0111000080010000000400000008CDDF0AFB691F41BA09018EF90FCD4000000060DFDEFA3FD37A14D5FA691F4183A048810610CD4072BC746BD03B104008CDDF0AFB691F41BA09018EF90FCD4072BC746BD03B104008CDDF0AFB691F41BA09018EF90FCD4000000060DFDEFA3F01110000800100000004000000D37A14D5FA691F4183A048810610CD4000000060DFDEFA3F08CDDF0AFB691F41BA09018EF90FCD4000000060DFDEFA3F35F609BBF4691F4137C9F7AD900FCD4000000060DFDEFA3FD37A14D5FA691F4183A048810610CD4000000060DFDEFA3F01110000800100000004000000005CC17ADD691F4100A0C09E2915CD4000000060DFDEFA3FD37A14D5FA691F4183A048810610CD4000000060DFDEFA3F35F609BBF4691F4137C9F7AD900FCD4000000060DFDEFA3F005CC17ADD691F4100A0C09E2915CD4000000060DFDEFA3F01110000800100000004000000EF68F0F8E3691F410436E5809515CD4000000060DFDEFA3FEFB06D03FB691F4104F663830910CD4000000060DFDEFA3FD37A14D5FA691F4183A048810610CD4000000060DFDEFA3FEF68F0F8E3691F410436E5809515CD4000000060DFDEFA3F01110000800100000004000000EF68F0F8E3691F410436E5809515CD4000000060DFDEFA3FD37A14D5FA691F4183A048810610CD4000000060DFDEFA3F005CC17ADD691F4100A0C09E2915CD4000000060DFDEFA3FEF68F0F8E3691F410436E5809515CD4000000060DFDEFA3F0111000080010000000400000035F609BBF4691F4137C9F7AD900FCD4072BC746BD03B104008CDDF0AFB691F41BA09018EF90FCD4072BC746BD03B1040D37A14D5FA691F4183A048810610CD4072BC746BD03B104035F609BBF4691F4137C9F7AD900FCD4072BC746BD03B10400111000080010000000400000035F609BBF4691F4137C9F7AD900FCD4072BC746BD03B1040D37A14D5FA691F4183A048810610CD4072BC746BD03B1040005CC17ADD691F4100A0C09E2915CD4072BC746BD03B104035F609BBF4691F4137C9F7AD900FCD4072BC746BD03B104001110000800100000004000000D37A14D5FA691F4183A048810610CD4072BC746BD03B1040EFB06D03FB691F4104F663830910CD4072BC746BD03B1040EF68F0F8E3691F410436E5809515CD4072BC746BD03B1040D37A14D5FA691F4183A048810610CD4072BC746BD03B104001110000800100000004000000005CC17ADD691F4100A0C09E2915CD4072BC746BD03B1040D37A14D5FA691F4183A048810610CD4072BC746BD03B1040EF68F0F8E3691F410436E5809515CD4072BC746BD03B1040005CC17ADD691F4100A0C09E2915CD4072BC746BD03B1040'

trying to read it using sharply:

from shapely import wkb

wkb.loads(geom) (or wkb.loads(geom, hex=True)

gives the error:

ParseException: Unknown WKB type 16

---> 3 wkb.loads(geom, hex=True)

--> 325 return lib.from_wkb(geometry, invalid_handler, **kwargs)

GEOSException: ParseException: Unknown WKB type 16

spark :

df=spark.sql("select distinct st_geomFromWKB(((hex(unbase64(after.geom.wkb))))) as wkb from .....")

org.locationtech.jts.io.ParseException: Unknown WKB type 16

at org.locationtech.jts.io.WKBReader.readGeometry(WKBReader.java:282)

at org.locationtech.jts.io.WKBReader.read(WKBReader.java:191)

at org.locationtech.jts.io.WKBReader.read(WKBReader.java:159)

at org.apache.sedona.core.formatMapper.FormatUtils.readWkb(FormatUtils.java:227)

any idea?

thanks

[Kaniz]

Hi @Amit Cahanovich​, The error message "Unknown WKB type 16" indicates that the WKB data you are trying to read has a geometry type that the library does not recognize.

WKB type 16 is not valid in the Simple Feature Access (SFA) standard, the most widely used standard for representing spatial data in WKB format.

The WKB data may be in a non-standard format specific to the API you are using. In this case, you may need to consult the API documentation to determine how to parse the data correctly.

Another possibility is that the WKB data is corrupted or incomplete. You can try validating the WKB data using an OGC Simple Features for SQL (SFSQL) tool to check if it is well-formed.

If the WKB data is valid and you cannot parse it, you can try using a different library or tool to parse the data. For example, you can use GeoPandas or PySAL to read the WKB data.

[amitca71 · Accepted solution]

Hi @Kaniz Fatma​ 

you are right. the type is not supported.

so far, i couldnt find any library that does support types 15 and 16...

thanks alot!!

Amit