topic Re: Error org.apache.spark.SparkSQLException: Unsupported type TIMESTAMP_WITH_TIMEZONE in Data Engineering

Error org.apache.spark.SparkSQLException: Unsupported type TIMESTAMP_WITH_TIMEZONE

KaushikMaji — Fri, 03 Feb 2023 12:17:27 GMT

Hi Guys,

I am trying to load data from a custom JDBC data source like below. I am getting this error, in spite of specifying "customSchema" attribute on the timestamp column. Do you know how to resolve this?

material_master = spark.read \

.format("jdbc") \

.option("url","<my jdbc url>") \

.option("driver", "<my jdbc driver>") \

.option("customSchema", "timestamp STRING") \

.option("query","SELECT CONO, STAT, ITNO, timestamp FROM MITMAS LIMIT 10") \

.load()

Re: Error org.apache.spark.SparkSQLException: Unsupported type TIMESTAMP_WITH_TIMEZONE

RaghavendraY — Fri, 03 Feb 2023 14:14:15 GMT

May try cast timestamp, like "SELECT CONO, STAT, ITNO, cast(timestamp as varchar) FROM MITMAS LIMIT 10"

Re: Error org.apache.spark.SparkSQLException: Unsupported type TIMESTAMP_WITH_TIMEZONE

Anonymous — Mon, 10 Apr 2023 08:59:30 GMT

Hi @Kaushik Maji

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!