spark databricks jdbc driver integration return rows having column names as values

manohar3
New Contributor III

‎01-29-2024 08:01 AM

Hi all,

i am using below to code to query table but query returns rows having column names as values

spark.read
   .format("jdbc")
   .option("url", "jdbc:databricks://acme.cloud.databricks.com:443/myschema;transportMode=http;ssl=1;httpPath=<httppath>;AuthMech=3;UID=token;PWD=<token>;")
   .option("dbtable", "mytable")
   .load().show()

+---+---------+
|key|value|
+---+---------+
|key|value|
|key|value|
|key|value|
|key|value|

jdbc driver version is 2.6.36

note that odbc driver works but i dont support struct, map, array as column names .


 

0 Kudos

manohar3
New Contributor III

‎01-29-2024 09:25 AM

any idea what causing this? please note that above query i am running from standalone spark application also when i run above in databricks notebook i see expected result

0 Kudos

manohar3
New Contributor III

‎01-30-2024 06:41 PM

This seems to be issue with spark and was able to fix issue by following posts

https://stackoverflow.com/questions/47020379/bigquery-simba-jdbc-error-with-spark

https://stackoverflow.com/questions/68013347/how-to-register-a-jdbc-spark-dialect-in-python/71913245...

fix is to update function in JdbcDialects.scala

def quoteIdentifier(colName: String): String = {
colName
}

 

https://github.com/apache/spark/blob/ce5ddad990373636e94071e7cef2f31021add07b/sql/core/src/main/scal...

View solution in original post

0 Kudos