cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
jonathan-dufaul
by • Valued Contributor
  • 1436 Views
  • 1 replies
  • 1 kudos

How do I specify column types when writing to a MSSQL server using the JDBC driver (

I have a pyspark dataframe that I'm writing to an on-prem MSSQL server--it's a stopgap while we convert data warehousing jobs over to databricks. The processes that use those tables in the on-prem server rely on the tables maintaining the identical s...

Data Engineering
Reply
  • 1436 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
dasanro
New Contributor II
  • 1 kudos

It's happenging to me too!Did you find any solution @jonathan-dufaul  ?Thanks!!

  • 1 kudos
AB_MN
by • New Contributor III
  • 6090 Views
  • 4 replies
  • 1 kudos

Resolved! Read data from Azure SQL DB

I am trying to read data into a dataframe from Azure SQL DB, using jdbc. Here is the code I am using.driver = "com.microsoft.sqlserver.jdbc.SQLServerDriver"   database_host = "server.database.windows.net" database_port = "1433" database_name = "dat...

Data Engineering
Reply
  • 6090 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
AB_MN
New Contributor III
  • 1 kudos

That did the trick. Thank you!

  • 1 kudos
3 More Replies
dng
by • New Contributor III
  • 6516 Views
  • 6 replies
  • 10 kudos

Databricks JDBC Driver v2.6.29 Cloud Fetch failing for Windows Operating System

Hi everyone, I've been stuck for the past two days on this issue with my Databricks JDBC driver and I'm hoping someone can give me more insight into how to troubleshoot. I am using the Databricks JDBC driver in RStudio and the connection was working ...

Data Engineering
Reply
  • 6516 Views
  • 6 replies
  • 10 kudos
View Replies
Latest Reply
Prabakar
Databricks Employee
  • 10 kudos

@Debbie Ng​ From your message I see there was a windows update and this failure started. based on the conversation you tried latest version of the driver and still you face the problem. I believe this is something related to the Java version compatib...

  • 10 kudos
5 More Replies
brian_0305
by • New Contributor II
  • 3921 Views
  • 3 replies
  • 2 kudos

Use JDBC connect to databrick default cluster and read table into pyspark dataframe. All the column turned into same as column name

I used code like below to Use JDBC connect to databrick default cluster and read table into pyspark dataframeurl = 'jdbc:databricks://[workspace domain]:443/default;transportMode=http;ssl=1;AuthMech=3;httpPath=[path];AuthMech=3;UID=token;PWD=[your_ac...

error
Data Engineering
Reply
  • 3921 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

@yu zhang​ :It looks like the issue with the first code snippet you provided is that it is not specifying the correct query to retrieve the data from your database.When using the load() method with the jdbc data source, you need to provide a SQL quer...

  • 2 kudos
2 More Replies
hfrid
by • New Contributor II
  • 5400 Views
  • 1 replies
  • 2 kudos

JDBC connector seems to be a bottleneck when trying to insert dataframe to Azure SQL Server

Hi! I am inserting a pyspark dataframe to Azure sql server and it takes a very long time. The database is a s4 but my dataframe that is 17 million rows and 30 columns takes up to 50 minutes to insert.Is there a way to significantly speed this up? I a...

Data Engineering
Reply
  • 5400 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

@Hjalmar Friden​ :There are several ways to improve the performance of inserting data into Azure SQL Server using JDBC connector:Increase the batch size: By default, the JDBC connector sends data in batches of 1000 rows at a time. You can increase th...

  • 2 kudos
pandu
by • New Contributor II
  • 2293 Views
  • 2 replies
  • 3 kudos

connect to Oracle database using JDBC and perform merge condition

I would like to connect to oracle database using JDBC driver and write a code to perform merge condition using python.

Data Engineering
Reply
  • 2293 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Vartika
Databricks Employee
  • 3 kudos

Hi @Venkata Krishna Jonnalagadda​ Hope you are well.Just checking in. If @John Lourdu​'s answer helped, would you let us know and mark the answer as best? If not, would you be happy to give us more information?Thanks!

  • 3 kudos
1 More Replies
haggholm
by • New Contributor
  • 2378 Views
  • 2 replies
  • 1 kudos

Resolved! Query with ORDER BY fails with HiveThriftServerError "requirement failed: Subquery … has not finished"

Using ODBC or JDBC to read from a table fails when I attempt to use an ORDER BY clause. In one sample case, I have a fairly small table (just 1946 rows).select * from some_table order by some_fieldResult:java.lang.IllegalArgumentException: requiremen...

Data Engineering
Reply
  • 2378 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @petter@hightouch.com Petter​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it doe...

  • 1 kudos
1 More Replies
RamyaN
by • New Contributor II
  • 3045 Views
  • 2 replies
  • 3 kudos

How to read enum[] (enum of array) datatype from postgres using spark

We are trying to read a column which is enum of array datatype from postgres as string datatype to target. We could able to achieve this by expilcitly using concat function while extracting like belowval jdbcDF3 = spark.read .format("jdbc") .option(...

Data Engineering
Reply
  • 3045 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 3 kudos

You can try custom schema for JDBC read.option("customSchema", "colname STRING")

  • 3 kudos
1 More Replies
jneira
by • New Contributor III
  • 2024 Views
  • 2 replies
  • 2 kudos

"org.apache.hadoop.hive.ql.metadata.HiveException: at least one column must be specified for the table" non deterministic error in a `insert ... select ... ` clause

Hi, first of all thahks for your work in databricks sql.Unfortunately i am having a problem running insert-selects statements programatically using the jdbc driver.They all have the form:`insert into `mytable` select 1, 'foo', moreLiterals`The statem...

Data Engineering
Reply
  • 2024 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
jneira
New Contributor III
  • 2 kudos

thanks for the suggestion, could tell me more about how to check logs in the cluster?

  • 2 kudos
1 More Replies
huyd
by • New Contributor III
  • 1190 Views
  • 0 replies
  • 4 kudos

Optimizing a batch load process, reading with the JDBC driver

I am doing a batch load, using the JDBC driver from a database table. I am noticing in Sparkui, that there is both memory and disk spill, but only on one executor. I am also, noticing that when trying to use the JDBC parallel read, it seems to run sl...

Data Engineering
Reply
  • 1190 Views
  • 0 replies
  • 4 kudos
sriramkumar
by • New Contributor II
  • 1246 Views
  • 2 replies
  • 1 kudos

Reasons for new Databricks driver

What are the reasons behind Databricks going for their own driver? What differences are made when switching between the previous Spark driver and the new Databricks driver?Is there any specific document I can look at or just the release notes?Also, w...

Data Engineering
Reply
  • 1246 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hey @Sriramkumar Thamizharasan​ Hope all is well! Just wanted to check in if you were able to resolve your issue would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from...

  • 1 kudos
1 More Replies
sriramkumar
by • New Contributor II
  • 2955 Views
  • 3 replies
  • 0 kudos

New Databricks Driver gives SQLNonTransientConnectionException when trying to connect to Databricks Instance

import com.databricks.client.jdbc.DataSource;   import java.sql.*;   public class testDatabricks { public static void main(String[] args) throws SQLException { String dbUrl = "jdbc:databricks://<hostname>:443;HttpPath=<HttpPath>;"; // Cop...

Data Engineering
Reply
  • 2955 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Atanu
Databricks Employee
  • 0 kudos

This looks like due to maintenance on US . Are you still facing the issue @Sriramkumar Thamizharasan​ Is your workspace on eastus and eastus2 ?

  • 0 kudos
2 More Replies
Vamsee
by • New Contributor II
  • 5930 Views
  • 4 replies
  • 4 kudos

Resolved! I am trying to connect to Databricks SQL using this JDBC driver. com.simba.spark.jdbc.Driver. I can download it and install it on my local laptop , but I need it to be available to maven repository so I can include the maven path in my project.

Please lett me know if the JDB driver is available in maven repository and what needs to be included in pom.xml to download it.

Data Engineering
Reply
  • 5930 Views
  • 4 replies
  • 4 kudos
View Replies
Latest Reply
BilalAslamDbrx
Databricks Employee
  • 4 kudos

@Vamsee krishna kanth Arcot​ good news, the driver is up on Maven: https://search.maven.org/artifact/com.databricks/databricks-jdbc

  • 4 kudos
3 More Replies
findinpath
by • Contributor
  • 5792 Views
  • 2 replies
  • 3 kudos

Databricks 2.6.25 JDBC driver can't create tables with `GENERATED` columns

I'm using the Databricks JDBC driver recently made available via Maven:https://mvnrepository.com/artifact/com.databricks/databricks-jdbc/2.6.25While trying to create a table with `GENERATED` columns I receive the following exception:Caused by: java.s...

Data Engineering
Reply
  • 5792 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
findinpath
Contributor
  • 3 kudos

I was under the impression that this has been recognised as a BUG and is being handled by Databricks.What do I need to do for reporting the issue officially as a BUG?

  • 3 kudos
1 More Replies
SCOR
by • New Contributor II
  • 2080 Views
  • 3 replies
  • 4 kudos

SparkJDBC42.jar Issue ?

Hi there!I am using the SparkJDBC42.jar in my Java application to use my delta lake tables , The connection is made through databricks sql endpoint in where I created a database and store in it my delta tables. I have a simple code to open connection...

Data Engineering
Reply
  • 2080 Views
  • 3 replies
  • 4 kudos
View Replies
Latest Reply
jose_gonzalez
Databricks Employee
  • 4 kudos

Hi @Seifeddine SNOUSSI​ ,Are you still having issue or you were able to resolve this issue? please let us know

  • 4 kudos
2 More Replies
Labels