cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Megan05
by • New Contributor III
  • 3387 Views
  • 5 replies
  • 4 kudos

Resolved! Out of Memory/Connection Lost When Writing to External SQL Server from Databricks Using JDBC Connection

I am working on writing a large amount of data from Databricks to an external SQL server using a JDB connection. I keep getting timeout errors/connection lost but digging deeper it appears to be a memory problem. I am wondering what cluster configura...

Data Engineering
Reply
  • 3387 Views
  • 5 replies
  • 4 kudos
View Replies
Latest Reply
hotrabattecom
New Contributor II
  • 4 kudos

Thanks for the answer. I am also get in this problem. Hotrabatt

  • 4 kudos
4 More Replies
LoiNguyen
by • New Contributor II
  • 15765 Views
  • 5 replies
  • 2 kudos

The authentication type 10 is not supported

I use below code to connect to postgresql. df = spark.read \ .jdbc("jdbc:postgresql://hostname:5432/dbname", "schema.table", properties={"user": "user", "password": "password"})\ .load() df.printSchema() However, I got the ...

Data Engineering
Reply
  • 15765 Views
  • 5 replies
  • 2 kudos
View Replies
Latest Reply
simboss
New Contributor II
  • 2 kudos

But how are we going to do this for those who use Windows?

  • 2 kudos
4 More Replies
Aj2
by • New Contributor III
  • 10139 Views
  • 4 replies
  • 1 kudos

Resolved! How to connect to DB2-AS400?

What are the steps needed to connect to a DB2-AS400 source to pull data to lake using Databricks? I believe it requires establishing a jdbc connection, but I couldnot find much details online

Data Engineering
Reply
  • 10139 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Ajay Menon​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 1 kudos
3 More Replies
jonasmin
by • New Contributor III
  • 8502 Views
  • 7 replies
  • 2 kudos

Error while establishing JDBC connection to Azure databricks via HTTP proxy

I am using the databricks JDBC driver (https://databricks.com/spark/jdbc-drivers-download) to connect to Azure databricks.The connection needs to be routed through a HTTP proxy. I found parameters that can be configured for using the HTTP proxy:By pa...

databricks jdbc
Data Engineering
Reply
  • 8502 Views
  • 7 replies
  • 2 kudos
View Replies
Latest Reply
MS_Varma
New Contributor II
  • 2 kudos

Hi @Jonas Minning​ , actually I am also having the same issue and when i looked into the driver related documentation I found that the driver currently only supports SOCKS proxies and I believe this is the reason why we are getting this error. So, I ...

  • 2 kudos
6 More Replies
Joao_DE
by • New Contributor III
  • 2307 Views
  • 4 replies
  • 0 kudos

JDBC connection

Hi everyone! I have a question. For a project I need to establish a jdbc connection using spark.read. My question is when does the connection is deleted. That is because I will read multiple tables from that database, so if I could just create a conn...

Data Engineering
Reply
  • 2307 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
Joao_DE
New Contributor III
  • 0 kudos

Hi Vidula!I haven´t figure out a solution yet, so any help would be appreciatedThank you!

  • 0 kudos
3 More Replies
elgeo
by • Valued Contributor II
  • 13448 Views
  • 7 replies
  • 4 kudos

Resolved! Invalid JDBC url

Hello. I am trying to establish a connection between DBeaver and Databricks. I followed the steps in DBeaver integration with Databricks | Databricks on AWS, but I get the following error while testing the connection: Could anyone provide any insight...

jdbc_url_error
Data Engineering
Reply
  • 13448 Views
  • 7 replies
  • 4 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @ELENI GEORGOUSI​ Glad to hear! It's a request that mark an answer as best.Thanks...

  • 4 kudos
6 More Replies
Trey
by • New Contributor III
  • 1885 Views
  • 2 replies
  • 3 kudos

Where do you usually store and manage "JDBC credentials" to use on databricks notebook?

Hi all,I would like to improve the way I use JDBC credenditial information (ID/PW, host, port, etc)Where do you guys usually store and use the jdbc credentials?Thanks for your help in advance!

Data Engineering
Reply
  • 1885 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Kwangwon Yi​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...

  • 3 kudos
1 More Replies
dataAllMyLife
by • New Contributor
  • 1108 Views
  • 1 replies
  • 0 kudos

JDBC Connection closes between 'stmt.execute( ... ) and stmt.executeQuery( ... )

I'm running a Java application that registers a CSV table with HIVE and then checks the number of rows imported. Its done in several steps.:Statement stmt = con.createStatement();....stmt.execute( "CREATE TABLE ( <definition> < > );.....ResultSet rs...

Data Engineering
Reply
  • 1108 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Noopur_Nigam
Databricks Employee
  • 0 kudos

@Reto Matter​  Are you running a jar job or using dbconnect to run java code? Please provide how are you trying to make a connection and full exception stack trace.

  • 0 kudos
mortenhaga
by • Contributor
  • 2622 Views
  • 2 replies
  • 2 kudos

Resolved! Importing python function with spark.read.jdbc in to Repos

Hi all!Before we used Databricks Repos we used the run magic to run various utility python functions from one notebook inside other notebooks, fex like reading from a jdbc connections. We now plan to switch to repos to utilize the fantastic CI/CD pos...

image.png image
Data Engineering
Reply
  • 2622 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
mortenhaga
Contributor
  • 2 kudos

Thats...odd. I was sure I had tried that, but now it works somehow. I guess it has to be that now I did it with double quotation marks. Thanks anyway! Works like a charm.

  • 2 kudos
1 More Replies
sriramkumar
by • New Contributor II
  • 3061 Views
  • 3 replies
  • 0 kudos

New Databricks Driver gives SQLNonTransientConnectionException when trying to connect to Databricks Instance

import com.databricks.client.jdbc.DataSource;   import java.sql.*;   public class testDatabricks { public static void main(String[] args) throws SQLException { String dbUrl = "jdbc:databricks://<hostname>:443;HttpPath=<HttpPath>;"; // Cop...

Data Engineering
Reply
  • 3061 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Atanu
Databricks Employee
  • 0 kudos

This looks like due to maintenance on US . Are you still facing the issue @Sriramkumar Thamizharasan​ Is your workspace on eastus and eastus2 ?

  • 0 kudos
2 More Replies
Quan
by • New Contributor III
  • 18706 Views
  • 8 replies
  • 5 kudos

Resolved! How to properly load Unicode (UTF-8) characters from table over JDBC connection using Simba Spark Driver

Hello all, I'm trying to pull table data from databricks tables that contain foreign language characters in UTF-8 into an ETL tool using a JDBC connection. I'm using the latest Simba Spark JDBC driver available from the Databricks website.The issue i...

Data Engineering
Reply
  • 18706 Views
  • 8 replies
  • 5 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 5 kudos

Can you try setting UseUnicodeSqlCharacterTypes=1 in the driver, and also make sure 'file.encoding' is set to UTF-8 in jvm and see if the issue still persists?

  • 5 kudos
7 More Replies
MohitAnchlia
by • New Contributor II
  • 1451 Views
  • 0 replies
  • 0 kudos

Accessing databricks from Presto SSQL

What's the best way to federate a query to delta lake or the databricks from presto sql without having to create external tables? PrestoSQL doesn't have access to S3. Can PrestoSQL be configured with jdbc driver or plugin?

Data Engineering
Reply
  • 1451 Views
  • 0 replies
  • 0 kudos
Labels