cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
swzzzsw
by • New Contributor III
  • 4949 Views
  • 6 replies
  • 0 kudos

Resolved! SQLServerException: deadlock

I'm using databricks to connect to a SQL managed instance via JDBC. SQL operations I need to perform include DELETE, UPDATE, and simple read and write. Since spark syntax only handles simple read and write, I had to open SQL connection using Scala an...

image.png
Data Engineering
Reply
  • 4949 Views
  • 6 replies
  • 0 kudos
View Replies
Latest Reply
Panda
Valued Contributor
  • 0 kudos

@swzzzsw Since you are performing database operations, to reduce the chances of deadlocks, make sure to wrap your SQL operations inside transactions using commit and rollback.Another approachs to consider is adding retry logic or using Isolation Leve...

  • 0 kudos
5 More Replies
Tico23
by • Contributor
  • 14690 Views
  • 12 replies
  • 10 kudos

Connecting SQL Server (on-premise) to Databricks via jdbc:sqlserver

Is it possible to connect to SQL Server on-premise (Not Azure) from Databricks?I tried to ping my virtualbox VM (with Windows Server 2022) from within Databricks and the request timed out.%sh   ping 122.138.0.14This is what my connection might look l...

Data Engineering
Reply
  • 14690 Views
  • 12 replies
  • 10 kudos
View Replies
Latest Reply
BharathKumarS
New Contributor II
  • 10 kudos

I tried to connect to localhost sql server through databricks community edition, but it failed. I have created an IP rule on port 1433 allowed inbound connection from all public network, but still didn't connect. I tried locally using python its work...

  • 10 kudos
11 More Replies
User16752244127
by • Contributor
  • 978 Views
  • 1 replies
  • 0 kudos

Can DLT be used to ingest from JDBC and/or and on-prem database?

Data Engineering
Reply
  • 978 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
lucasrocha
Databricks Employee
  • 0 kudos

Hello @User16752244127 , I hope this message finds you well. Delta Live Tables supports loading data from any data source supported by Databricks. You can find the datasources supported here Connect to data sources, and JDBC is one of them. You can a...

  • 0 kudos
Hardy
by • New Contributor III
  • 7169 Views
  • 5 replies
  • 5 kudos

The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption

I am trying to connect to SQL through JDBC from databricks notebook. (Below is my notebook command)val df = spark.read.jdbc(jdbcUrl, "[MyTableName]", connectionProperties) println(df.schema)When I execute this command, with DBR 10.4 LTS it works fin...

Data Engineering
Reply
  • 7169 Views
  • 5 replies
  • 5 kudos
View Replies
Latest Reply
DBXC
Contributor
  • 5 kudos

Try to add the following parameters to your SQL connection string. It fixed my problem for 13.X and 12.X;trustServerCertificate=true;hostNameInCertificate=*.database.windows.net; 

  • 5 kudos
4 More Replies
dprutean
by • New Contributor III
  • 11207 Views
  • 4 replies
  • 4 kudos

Resolved! JDBC Driver support for OpenJDK 17

Connecting to Databricks using OpenJDK 17 I got the exception below. Are there any plans to fix the driver for OpenJDK17?java.sql.SQLException: [Databricks][DatabricksJDBCDriver](500540) Error caught in BackgroundFetcher. Foreground thread ID: 44. Ba...

Data Engineering
Reply
  • 11207 Views
  • 4 replies
  • 4 kudos
View Replies
Latest Reply
ameyabapat
New Contributor II
  • 4 kudos

I still see the above error with databricks jdbc driver 2.6.33. Anyone aware of fix available either in  driver or java? 

  • 4 kudos
3 More Replies
User16765131552
by • Contributor III
  • 6655 Views
  • 5 replies
  • 1 kudos

How to register a JDBC Spark dialect in Python?

I am trying to read from a databricks table. I have used the url from a cluster in the databricks. I am getting this error: java.sql.SQLDataException: [Simba][JDBC](10140) Error converting value to int.After these statements:jdbcConnUrl= "jdbc:spark:...

Data Engineering
Reply
  • 6655 Views
  • 5 replies
  • 1 kudos
View Replies
Latest Reply
KKDataEngineer
New Contributor III
  • 1 kudos

is there a solution for this?

  • 1 kudos
4 More Replies
siddharthk
by • New Contributor II
  • 1781 Views
  • 2 replies
  • 2 kudos

Resolved! Reduce downtime of Postgres table - JDBC overwrite job

I want to overwrite a Postgresql table transactionStats which is used by the customer facing dashboards.This table needs to be updated every 30 mins. I am writing a AWS Glue Spark job via JDBC connection to perform this operation.Spark dataframe writ...

Data Engineering
Reply
  • 1781 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Siddharth Kanojiya​ We haven't heard from you since the last response from @werners (Customer)​ . Kindly share the information with us, and in return, we will provide you with the necessary solution.Thanks and Regards

  • 2 kudos
1 More Replies
jonasmin
by • New Contributor III
  • 8863 Views
  • 7 replies
  • 2 kudos

Error while establishing JDBC connection to Azure databricks via HTTP proxy

I am using the databricks JDBC driver (https://databricks.com/spark/jdbc-drivers-download) to connect to Azure databricks.The connection needs to be routed through a HTTP proxy. I found parameters that can be configured for using the HTTP proxy:By pa...

databricks jdbc
Data Engineering
Reply
  • 8863 Views
  • 7 replies
  • 2 kudos
View Replies
Latest Reply
MS_Varma
New Contributor II
  • 2 kudos

Hi @Jonas Minning​ , actually I am also having the same issue and when i looked into the driver related documentation I found that the driver currently only supports SOCKS proxies and I believe this is the reason why we are getting this error. So, I ...

  • 2 kudos
6 More Replies
haraldh
by • New Contributor II
  • 1709 Views
  • 1 replies
  • 2 kudos

Databericks JDBC driver connection pooling support

When using Camel JDBC with Databricks JDBC driver I get an error: Caused by: java.sql.SQLFeatureNotSupportedException: [Databricks][JDBC](10220) Driver does not support this optional feature.Is there any means to work around this limitation?

Data Engineering
Reply
  • 1709 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
swethaNandan
Databricks Employee
  • 2 kudos

Tools like SDI can connect to a generic JDBC source such as Databricks SQL Warehouse via the SDI Camel JDBC adapter. can you see  if these will help you https://help.sap.com/docs/HANA_SMART_DATA_INTEGRATION/7952ef28a6914997abc01745fef1b607/1247c9518...

  • 2 kudos
Alexander1
by • New Contributor III
  • 10475 Views
  • 5 replies
  • 1 kudos

Resolved! Databricks JDBC/ODBC write batch size

I have spent way too much time to find a solution to the problem of efficiently writing data to Databricks via JDBC/ODBC. I have looked into countless docs, blogs and repos and I cannot find one example where someone is setting some kind of batch/bul...

Data Engineering
Reply
  • 10475 Views
  • 5 replies
  • 1 kudos
View Replies
Latest Reply
Alexander1
New Contributor III
  • 1 kudos

@Vidula Khanna​ yes, have done so. thanks.

  • 1 kudos
4 More Replies
Joao_DE
by • New Contributor III
  • 2823 Views
  • 3 replies
  • 1 kudos

Information about stoping JDBC connection

Hi everyone!I would like to know how spark stops the connection when reading from a sql database using the JDBC format.Also, if there is a way to check when the connection is active or manually stop it, I also would like to know.Thank you in advance!

Data Engineering
Reply
  • 2823 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @João Peixoto​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so ...

  • 1 kudos
2 More Replies
pandu
by • New Contributor II
  • 2592 Views
  • 2 replies
  • 3 kudos

connect to Oracle database using JDBC and perform merge condition

I would like to connect to oracle database using JDBC driver and write a code to perform merge condition using python.

Data Engineering
Reply
  • 2592 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Vartika
Databricks Employee
  • 3 kudos

Hi @Venkata Krishna Jonnalagadda​ Hope you are well.Just checking in. If @John Lourdu​'s answer helped, would you let us know and mark the answer as best? If not, would you be happy to give us more information?Thanks!

  • 3 kudos
1 More Replies
haggholm
by • New Contributor
  • 2623 Views
  • 2 replies
  • 1 kudos

Resolved! Query with ORDER BY fails with HiveThriftServerError "requirement failed: Subquery … has not finished"

Using ODBC or JDBC to read from a table fails when I attempt to use an ORDER BY clause. In one sample case, I have a fairly small table (just 1946 rows).select * from some_table order by some_fieldResult:java.lang.IllegalArgumentException: requiremen...

Data Engineering
Reply
  • 2623 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @petter@hightouch.com Petter​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it doe...

  • 1 kudos
1 More Replies
Skesaram
by • New Contributor II
  • 1341 Views
  • 1 replies
  • 0 kudos

Need help to connect to local DB from Data bricks

jdbcHostname="478"jdbcPort=1433jdbcDatabase="Onprem_AzureDB"jdbcUsername="upendra"jdbcPassword="upendrakumar"jdbcDriver="com.microsoft.sqlserver.jdbc.SQLServerDriver"jdbcUrl=f"jdbc:sqlserver://{jdbcHostname}:{jdbcPort};databaseName={jdbcDatabase};use...

Data Engineering
Reply
  • 1341 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Debayan
Databricks Employee
  • 0 kudos

Hi, Could you please verify the network connectivity from Databricks to the SQL server? Please make sure SQL:port is allowed in your firewall rules or security groups.

  • 0 kudos
Fred_F
by • New Contributor III
  • 7626 Views
  • 5 replies
  • 5 kudos

JDBC connection timeout on workflow cluster

Hi there,​I've a batch process configured in a workflow which fails due to a jdbc timeout on a Postgres DB.​I checked the JDBC connection configuration and it seems to work when I query a table and doing a df.show() in the process and it displays th...

Data Engineering
Reply
  • 7626 Views
  • 5 replies
  • 5 kudos
View Replies
Latest Reply
RKNutalapati
Valued Contributor
  • 5 kudos

HI @Fred Foucart​ ,The above code looks good to me. Can you try with below code as well.spark.read\  .format("jdbc") \  .option("url", f"jdbc:postgresql://{host}/{database}") \  .option("driver", "org.postgresql.Driver") \  .option("user", username) ...

  • 5 kudos
4 More Replies
Labels