cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

How to register a JDBC Spark dialect in Python?

User16765131552
Contributor III

I am trying to read from a databricks table. I have used the url from a cluster in the databricks. I am getting this error:

 java.sql.SQLDataException: [Simba][JDBC](10140) Error converting value to int.

After these statements:

jdbcConnUrl= "jdbc:spark://adb....."
testquery="(select * from db.table limit 3)"
testdf=spark.read.format("jdbc").option("url", jdbcConnUrl).option("dbtable", testquery).option("fetchsize", "10000").load()
testdf.show()

I have come across all Scala solutions for this issue but I am using python. I want a python equivalent of this code:

import org.apache.spark.sql.jdbc.{JdbcDialect, JdbcDialects}
JdbcDialects.registerDialect(new JdbcDialect() {
override def canHandle(url: String): Boolean = url.toLowerCase.startsWith("jdbc:spark:")
override
def quoteIdentifier(column: String): String = column
})

6 REPLIES 6

Kaniz
Community Manager
Community Manager

Hi @ User16217864942579505928! My name is Kaniz, and I'm a technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers on the Forum have an answer to your questions first. Or else I will follow up shortly with a response.

imstwz1
New Contributor II

Hi @Brad Powell​, Were you able to solve this issue in Python. I'm also struck with this issue in python, solution is available only in Scala and need this solution in python. Could you please help me in solving this issue. Thanks

Meghala
Valued Contributor II

It was helpfull thank you​

Yadu
New Contributor II

Hi @Brad Powell​ / @Kaniz Fatma​ / @S Meghala​ - Any update on this?

KKDataEngineer
New Contributor III

is there a solution for this?

@Kaniz 

I was able to solve this 

  •  Add this code into a simple scala class object method 
  • package it into a JAR file
  • now install this JAR file on the cluster where you execute the JDBC code.
  • add the below line of code  before executing the JDBC code in you pyspark code. this will execute that class and method from scala class in your JVM directly.
    spark.sparkContext._jvm.<scalaclass fully qualified>.<method>
     
Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.