sedona/shapely error Unknown WKB type 16

amitca71 — Sun, 23 Apr 2023 19:21:47 GMT

Hi,

i stream data from postgis to s3 using debezium. postgis->debezium->s3->spark(databricks)

once read it i decode it and i can see that the binary representation is similiar to what i have in postgis, on a wkb formated column.

once i try to read it either using sedona or shapely, i get same error (Unknown WKB type 16 ).

the following is how i read the data from s3

this is how i read the wkb column:

select distinct hex(unbase64(after.geom.wkb)) as geom from ....

the result is:

geom ='01100000A0AF0E0000140000000111000080010000000400000035F609BBF4691F4137C9F7AD900FCD4000000060DFDEFA3F08CDDF0AFB691F41BA09018EF90FCD4000000060DFDEFA3F08CDDF0AFB691F41BA09018EF90FCD4072BC746BD03B104035F609BBF4691F4137C9F7AD900FCD4000000060DFDEFA3F0111000080010000000400000035F609BBF4691F4137C9F7AD900FCD4000000060DFDEFA3F08CDDF0AFB691F41BA09018EF90FCD4072BC746BD03B104035F609BBF4691F4137C9F7AD900FCD4072BC746BD03B104035F609BBF4691F4137C9F7AD900FCD4000000060DFDEFA3F01110000800100000004000000005CC17ADD691F4100A0C09E2915CD4000000060DFDEFA3F35F609BBF4691F4137C9F7AD900FCD4000000060DFDEFA3F35F609BBF4691F4137C9F7AD900FCD4072BC746BD03B1040005CC17ADD691F4100A0C09E2915CD4000000060DFDEFA3F01110000800100000004000000005CC17ADD691F4100A0C09E2915CD4000000060DFDEFA3F35F609BBF4691F4137C9F7AD900FCD4072BC746BD03B1040005CC17ADD691F4100A0C09E2915CD4072BC746BD03B1040005CC17ADD691F4100A0C09E2915CD4000000060DFDEFA3F01110000800100000004000000EF68F0F8E3691F410436E5809515CD4000000060DFDEFA3F005CC17ADD691F4100A0C09E2915CD4000000060DFDEFA3F005CC17ADD691F4100A0C09E2915CD4072BC746BD03B1040EF68F0F8E3691F410436E5809515CD4000000060DFDEFA3F01110000800100000004000000EF68F0F8E3691F410436E5809515CD4000000060DFDEFA3F005CC17ADD691F4100A0C09E2915CD4072BC746BD03B1040EF68F0F8E3691F410436E5809515CD4072BC746BD03B1040EF68F0F8E3691F410436E5809515CD4000000060DFDEFA3F01110000800100000004000000EFB06D03FB691F4104F663830910CD4000000060DFDEFA3FEF68F0F8E3691F410436E5809515CD4000000060DFDEFA3FEF68F0F8E3691F410436E5809515CD4072BC746BD03B1040EFB06D03FB691F4104F663830910CD4000000060DFDEFA3F01110000800100000004000000EFB06D03FB691F4104F663830910CD4000000060DFDEFA3FEF68F0F8E3691F410436E5809515CD4072BC746BD03B1040EFB06D03FB691F4104F663830910CD4072BC746BD03B1040EFB06D03FB691F4104F663830910CD4000000060DFDEFA3F01110000800100000004000000D37A14D5FA691F4183A048810610CD4000000060DFDEFA3FEFB06D03FB691F4104F663830910CD4000000060DFDEFA3FEFB06D03FB691F4104F663830910CD4072BC746BD03B1040D37A14D5FA691F4183A048810610CD4000000060DFDEFA3F01110000800100000004000000D37A14D5FA691F4183A048810610CD4000000060DFDEFA3FEFB06D03FB691F4104F663830910CD4072BC746BD03B1040D37A14D5FA691F4183A048810610CD4072BC746BD03B1040D37A14D5FA691F4183A048810610CD4000000060DFDEFA3F0111000080010000000400000008CDDF0AFB691F41BA09018EF90FCD4000000060DFDEFA3FD37A14D5FA691F4183A048810610CD4000000060DFDEFA3FD37A14D5FA691F4183A048810610CD4072BC746BD03B104008CDDF0AFB691F41BA09018EF90FCD4000000060DFDEFA3F0111000080010000000400000008CDDF0AFB691F41BA09018EF90FCD4000000060DFDEFA3FD37A14D5FA691F4183A048810610CD4072BC746BD03B104008CDDF0AFB691F41BA09018EF90FCD4072BC746BD03B104008CDDF0AFB691F41BA09018EF90FCD4000000060DFDEFA3F01110000800100000004000000D37A14D5FA691F4183A048810610CD4000000060DFDEFA3F08CDDF0AFB691F41BA09018EF90FCD4000000060DFDEFA3F35F609BBF4691F4137C9F7AD900FCD4000000060DFDEFA3FD37A14D5FA691F4183A048810610CD4000000060DFDEFA3F01110000800100000004000000005CC17ADD691F4100A0C09E2915CD4000000060DFDEFA3FD37A14D5FA691F4183A048810610CD4000000060DFDEFA3F35F609BBF4691F4137C9F7AD900FCD4000000060DFDEFA3F005CC17ADD691F4100A0C09E2915CD4000000060DFDEFA3F01110000800100000004000000EF68F0F8E3691F410436E5809515CD4000000060DFDEFA3FEFB06D03FB691F4104F663830910CD4000000060DFDEFA3FD37A14D5FA691F4183A048810610CD4000000060DFDEFA3FEF68F0F8E3691F410436E5809515CD4000000060DFDEFA3F01110000800100000004000000EF68F0F8E3691F410436E5809515CD4000000060DFDEFA3FD37A14D5FA691F4183A048810610CD4000000060DFDEFA3F005CC17ADD691F4100A0C09E2915CD4000000060DFDEFA3FEF68F0F8E3691F410436E5809515CD4000000060DFDEFA3F0111000080010000000400000035F609BBF4691F4137C9F7AD900FCD4072BC746BD03B104008CDDF0AFB691F41BA09018EF90FCD4072BC746BD03B1040D37A14D5FA691F4183A048810610CD4072BC746BD03B104035F609BBF4691F4137C9F7AD900FCD4072BC746BD03B10400111000080010000000400000035F609BBF4691F4137C9F7AD900FCD4072BC746BD03B1040D37A14D5FA691F4183A048810610CD4072BC746BD03B1040005CC17ADD691F4100A0C09E2915CD4072BC746BD03B104035F609BBF4691F4137C9F7AD900FCD4072BC746BD03B104001110000800100000004000000D37A14D5FA691F4183A048810610CD4072BC746BD03B1040EFB06D03FB691F4104F663830910CD4072BC746BD03B1040EF68F0F8E3691F410436E5809515CD4072BC746BD03B1040D37A14D5FA691F4183A048810610CD4072BC746BD03B104001110000800100000004000000005CC17ADD691F4100A0C09E2915CD4072BC746BD03B1040D37A14D5FA691F4183A048810610CD4072BC746BD03B1040EF68F0F8E3691F410436E5809515CD4072BC746BD03B1040005CC17ADD691F4100A0C09E2915CD4072BC746BD03B1040'

trying to read it using sharply:

from shapely import wkb

wkb.loads(geom) (or wkb.loads(geom, hex=True)

gives the error:

ParseException: Unknown WKB type 16

---> 3 wkb.loads(geom, hex=True)

--> 325 return lib.from_wkb(geometry, invalid_handler, **kwargs)

GEOSException: ParseException: Unknown WKB type 16

spark :

df=spark.sql("select distinct st_geomFromWKB(((hex(unbase64(after.geom.wkb))))) as wkb from .....")

org.locationtech.jts.io.ParseException: Unknown WKB type 16

at org.locationtech.jts.io.WKBReader.readGeometry(WKBReader.java:282)

at org.locationtech.jts.io.WKBReader.read(WKBReader.java:191)

at org.locationtech.jts.io.WKBReader.read(WKBReader.java:159)

at org.apache.sedona.core.formatMapper.FormatUtils.readWkb(FormatUtils.java:227)

any idea?

thanks

Re: sedona/shapely error Unknown WKB type 16

amitca71 — Thu, 27 Apr 2023 16:36:43 GMT

Hi @Kaniz Fatma

you are right. the type is not supported.

so far, i couldnt find any library that does support types 15 and 16...

thanks alot!!

Amit

topic sedona/shapely error Unknown WKB type 16 in Data Engineering

sedona/shapely error Unknown WKB type 16

Re: sedona/shapely error Unknown WKB type 16