topic spark databricks jdbc driver integration return rows having column names as values in Data Engineering

spark databricks jdbc driver integration return rows having column names as values

manohar3 — Mon, 29 Jan 2024 16:01:59 GMT

Hi all,

i am using below to code to query table but query returns rows having column names as values

spark.read
   .format("jdbc")
   .option("url", "jdbc:databricks://acme.cloud.databricks.com:443/myschema;transportMode=http;ssl=1;httpPath=<httppath>;AuthMech=3;UID=token;PWD=<token>;")
   .option("dbtable", "mytable")
   .load().show()

+---+---------+
|key|value|
+---+---------+
|key|value|
|key|value|
|key|value|
|key|value|

jdbc driver version is 2.6.36

note that odbc driver works but i dont support struct, map, array as column names .

Re: spark databricks jdbc driver integration return rows having column names as values

manohar3 — Mon, 29 Jan 2024 17:25:17 GMT

any idea what causing this? please note that above query i am running from standalone spark application also when i run above in databricks notebook i see expected result

Re: spark databricks jdbc driver integration return rows having column names as values

manohar3 — Wed, 31 Jan 2024 02:41:59 GMT

This seems to be issue with spark and was able to fix issue by following posts

https://stackoverflow.com/questions/47020379/bigquery-simba-jdbc-error-with-spark

https://stackoverflow.com/questions/68013347/how-to-register-a-jdbc-spark-dialect-in-python/71913245#71913245

fix is to update function in JdbcDialects.scala

def quoteIdentifier(colName: String): String = {
colName
}

https://github.com/apache/spark/blob/ce5ddad990373636e94071e7cef2f31021add07b/sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala#L171