cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

How to read enum[] (enum of array) datatype from postgres using spark

RamyaN
New Contributor II

We are trying to read a column which is enum of array datatype from postgres as string datatype to target. We could able to achieve this by expilcitly using concat function while extracting like below

val jdbcDF3 = spark.read

 .format("jdbc")

 .option("url", <jdbc url>)

 .option("query", "SELECT concat(colname) as colname FROM <tablename> ")

 .load()

Does spark not support reading array datatype as string by default without using concat function?

We are getting the "SQLException: Unsupported type ARRAY" error if we dont use concat function while extracting

2 REPLIES 2

Hubert-Dudek
Esteemed Contributor III

You can try custom schema for JDBC read

.option("customSchema", "colname STRING") 

RamyaN
New Contributor II

Tried.. No luck @Hubert Dudek​ 

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.