topic Re: How to read enum[] (enum of array) datatype from postgres using spark in Data Engineering

How to read enum[] (enum of array) datatype from postgres using spark

RamyaN — Thu, 05 Jan 2023 12:29:51 GMT

We are trying to read a column which is enum of array datatype from postgres as string datatype to target. We could able to achieve this by expilcitly using concat function while extracting like below

val jdbcDF3 = spark.read

.format("jdbc")

.option("url", <jdbc url>)

.option("query", "SELECT concat(colname) as colname FROM <tablename> ")

.load()

Does spark not support reading array datatype as string by default without using concat function?

We are getting the "SQLException: Unsupported type ARRAY" error if we dont use concat function while extracting

Re: How to read enum[] (enum of array) datatype from postgres using spark

Hubert-Dudek — Thu, 05 Jan 2023 13:12:26 GMT

You can try custom schema for JDBC read

.option("customSchema", "colname STRING")

Re: How to read enum[] (enum of array) datatype from postgres using spark

RamyaN — Fri, 06 Jan 2023 04:01:15 GMT

Tried.. No luck @Hubert Dudek