topic Re: How to register a JDBC Spark dialect in Python? in Data Engineering

How to register a JDBC Spark dialect in Python?

User16765131552 — Fri, 18 Jun 2021 20:13:14 GMT

I am trying to read from a databricks table. I have used the url from a cluster in the databricks. I am getting this error:

 java.sql.SQLDataException: [Simba][JDBC](10140) Error converting value to int.

After these statements:

jdbcConnUrl= "jdbc:spark://adb....."
testquery="(select * from db.table limit 3)"
testdf=spark.read.format("jdbc").option("url", jdbcConnUrl).option("dbtable", testquery).option("fetchsize", "10000").load()
testdf.show()

I have come across all Scala solutions for this issue but I am using python. I want a python equivalent of this code:

import org.apache.spark.sql.jdbc.{JdbcDialect, JdbcDialects}
JdbcDialects.registerDialect(new JdbcDialect() {
override def canHandle(url: String): Boolean = url.toLowerCase.startsWith("jdbc:spark:")
override
def quoteIdentifier(column: String): String = column
})

Re: How to register a JDBC Spark dialect in Python?

imstwz1 — Fri, 23 Dec 2022 06:47:22 GMT

Hi @Brad Powell, Were you able to solve this issue in Python. I'm also struck with this issue in python, solution is available only in Scala and need this solution in python. Could you please help me in solving this issue. Thanks

Re: How to register a JDBC Spark dialect in Python?

Meghala — Mon, 26 Dec 2022 14:30:45 GMT

It was helpfull thank you

Re: How to register a JDBC Spark dialect in Python?

Yadu — Fri, 30 Dec 2022 18:13:59 GMT

Hi @Brad Powell / @Kaniz Fatma / @S Meghala - Any update on this?

Re: How to register a JDBC Spark dialect in Python?

KKDataEngineer — Wed, 28 Jun 2023 10:20:12 GMT

is there a solution for this?

Re: How to register a JDBC Spark dialect in Python?

KKDataEngineer — Fri, 30 Jun 2023 18:45:28 GMT

@Retired_mod

I was able to solve this

Add this code into a simple scala class object method
package it into a JAR file
now install this JAR file on the cluster where you execute the JDBC code.
add the below line of code before executing the JDBC code in you pyspark code. this will execute that class and method from scala class in your JVM directly.
spark.sparkContext._jvm.<scalaclass fully qualified>.<method>

@User16765131552 @Yadu @imstwz1