cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Abel_Martinez
by • Contributor
  • 15494 Views
  • 9 replies
  • 10 kudos

Resolved! Why I'm getting connection timeout when connecting to MongoDB using MongoDB Connector for Spark 10.x from Databricks

I'm able to connect to MongoDB using org.mongodb.spark:mongo-spark-connector_2.12:3.0.2 and this code:df = spark.read.format("com.mongodb.spark.sql.DefaultSource").option("uri", jdbcUrl)It works well, but if I install last MongoDB Spark Connector ve...

Data Engineering
Reply
  • 15494 Views
  • 9 replies
  • 10 kudos
View Replies
Latest Reply
ravisharma1024
New Contributor II
  • 10 kudos

I was facing the same issue, now It is resolved, and thanks to @Abel_Martinez.I am using this like below code:df = spark.read.format("mongodb") \.option('spark.mongodb.read.connection.uri', "mongodb+srv://*****:*****@******/?retryWrites=true&w=majori...

  • 10 kudos
8 More Replies
MichelSmits
by • New Contributor III
  • 7651 Views
  • 4 replies
  • 3 kudos

Connection to on-prem database occasionally goes to the wrong server

We run a dbx job that uses a jdbc connection to three (SQL Server) on-prem databases. Very occasionally a connection will go to the wrong server.In the job log we see the error "com.microsoft.sqlserver.jdbc.SQLServerException: Login failed for user '...

Data Engineering
Reply
  • 7651 Views
  • 4 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Michel Smits​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...

  • 3 kudos
3 More Replies
Mani1800
by • New Contributor
  • 1970 Views
  • 2 replies
  • 0 kudos

I need to run Sql Update/Delete commands for a AWS RDS system.

I tried 'jdbc' connection to access the data from the RDS. I was able to read the data successfully but I need to do run some update queries. It seems the jdbc won't support update operation. I tried to make connection to my RDS mysql with host, user...

Data Engineering
Reply
  • 1970 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Manikandan Ramachandran​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear fro...

  • 0 kudos
1 More Replies
Geetha
by • New Contributor
  • 2902 Views
  • 2 replies
  • 0 kudos

Connection to Databricks through KNIME

I want to make a connection to Databricks with KNIME. To do this I am using "Create Databricks Environment" node. I have made the following configuration:1. Installed Databricks Simba JDBC driver 2. Made the necessary configuration in Create Databric...

Data Engineering
Reply
  • 2902 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Geethanjali Nataraj​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell...

  • 0 kudos
1 More Replies
Joao_DE
by • New Contributor III
  • 2980 Views
  • 3 replies
  • 1 kudos

Information about stoping JDBC connection

Hi everyone!I would like to know how spark stops the connection when reading from a sql database using the JDBC format.Also, if there is a way to check when the connection is active or manually stop it, I also would like to know.Thank you in advance!

Data Engineering
Reply
  • 2980 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @João Peixoto​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so ...

  • 1 kudos
2 More Replies
DeveloperAmarde
by • New Contributor
  • 1853 Views
  • 1 replies
  • 0 kudos

Connection to Collibra

Hi Team,I want to connect to collibra to fetch details from Collibra.Currently we are using username and password to connect.I want to know recommended practice to connect Collibra account from databricks notebook.

Data Engineering
Reply
  • 1853 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Debayan
Databricks Employee
  • 0 kudos

Hi, Could you please know if this helps. https://marketplace.collibra.com/listings/jdbc-driver-for-databricks/

  • 0 kudos
leandro
by • New Contributor
  • 2013 Views
  • 0 replies
  • 0 kudos

Connection from on-premise R sparklyr session to Databricks, " invalid method toDF for object 17/org.apache.spark.sql.DataFrameReader fields 0 selected 0"

Hello,I'm working with an on-premise R session and would like to connect to Databricks using sparklyr. RStudio server in this case is not an option.I would like to use jdbc. I tested RJDBC + DBI and can conect locally and perform operations. However,...

Data Engineering
Reply
  • 2013 Views
  • 0 replies
  • 0 kudos
mrmota
by • New Contributor
  • 2113 Views
  • 1 replies
  • 3 kudos

ssh connection with paramiko

I'm trying to use the notebook with python to access our server through ssh. But I am not able to connect through the paramiko library. I have already authorized our server's firewall to accept the databrick ip. Have you ever had a similar case? can ...

Data Engineering
Reply
  • 2113 Views
  • 1 replies
  • 3 kudos
View Replies
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 3 kudos

yes this can happen you have to whitelist that Port in the databricks configuration, there is one spark configuration that you can add to your cluster, that will whitelist your particular portas that configuration can only give by databricks, you can...

  • 3 kudos
elgeo
by • Valued Contributor II
  • 15668 Views
  • 7 replies
  • 4 kudos

Resolved! Invalid JDBC url

Hello. I am trying to establish a connection between DBeaver and Databricks. I followed the steps in DBeaver integration with Databricks | Databricks on AWS, but I get the following error while testing the connection: Could anyone provide any insight...

jdbc_url_error
Data Engineering
Reply
  • 15668 Views
  • 7 replies
  • 4 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @ELENI GEORGOUSI​ Glad to hear! It's a request that mark an answer as best.Thanks...

  • 4 kudos
6 More Replies
elgeo
by • Valued Contributor II
  • 1708 Views
  • 1 replies
  • 2 kudos

Databricks - DBeaver error

Hello experts. While trying to setup the connection between DBeaver and Databricks, we receive the following error:[Databricks][DatabricksJDBCDriver](700120) Host adb-xxxxxxxxxx.azuredatabricks.net cannot be resolved through DnsResolver com.databrick...

Data Engineering
Reply
  • 1708 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
elgeo
Valued Contributor II
  • 2 kudos

We identified the problem. It was due to the proxy that the hostname couldn't be resolved.

  • 2 kudos
Herkimer
by • New Contributor II
  • 1148 Views
  • 0 replies
  • 0 kudos

intermittent connection error

I am running dbsqlcli in windows 10. I have put together the attached cmd file to pull the identity column data from a series of our tables into individual CSVs so I can upload then to a PostgreSQL DB to do a comparison of each table to those in the ...

Data Engineering
Reply
  • 1148 Views
  • 0 replies
  • 0 kudos
nadia
by • New Contributor II
  • 1985 Views
  • 1 replies
  • 0 kudos

Resolved! Connection Databricks Postgresql

I use Databricks and I try to connect to posgresql via the following code"jdbcHostname = "xxxxxxx"jdbcDatabase = "xxxxxxxxxxxx"jdbcPort = "5432"username = "xxxxxxx"password = "xxxxxxxx"jdbcUrl = "jdbc:postgresql://{0}:{1}/{2}".format(jdbcHostname, jd...

Data Engineering
Reply
  • 1985 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Prabakar
Databricks Employee
  • 0 kudos

hi @Boumaza nadia​ Please check the Ganglia metrics for the cluster. This could be a scalability issue where cluster is overloading. This can happen due to a large partition not fitting into the given executor's memory. To fix this we recommend bump...

  • 0 kudos
Orianh
by • Valued Contributor II
  • 2707 Views
  • 2 replies
  • 2 kudos

Resolved! pyodbc read only connection.

Hey Guys, Is there a way to open pyodbc read only connection with simba spark driver? At the moment, I'm able to execute queries such as select , delete, insert into - basically every sql statement using pyodbc. I tried to open pyodbc connection but ...

Data Engineering
Reply
  • 2707 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 2 kudos

This readonly=True is working only on some drivers. Just create additional users with granted read-only permission.

  • 2 kudos
1 More Replies
RK_AV
by • New Contributor III
  • 1992 Views
  • 1 replies
  • 0 kudos

Databricks to Databricks connection

How do you connect to Azure Databricks instance from another Databricks instance? I needed to access (database) Views created in a Databricks instance from a Pyspark notebook running in another Databricks instance. Appreciate if anyone has any sample...

Data Engineering
Reply
  • 1992 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi there, @Venkata Ramakrishna Alvakonda​! My name is Piper, and I'm a moderator for the community. Thank you for your great question! Let's give the community a chance to respond first, and then we'll circle back around. If the community's response ...

  • 0 kudos
Rnmj
by • New Contributor III
  • 13787 Views
  • 3 replies
  • 6 kudos

ConnectException: Connection refused (Connection refused) This is often caused by an OOM error

I am trying to run a python code where a json file is flattened to pipe separated file . The code works with smaller files but for huge files of 2.4 GB I get below error:ConnectException: Connection refused (Connection refused)Error while obtaining a...

Data Engineering
Reply
  • 13787 Views
  • 3 replies
  • 6 kudos
View Replies
Latest Reply
Rnmj
New Contributor III
  • 6 kudos

Hi @Jose Gonzalez​ , @Werner Stinckens​  @Kaniz Fatma​ ,Thanks for your response .Appreciate a lot. The issue was in the code, it was a python /panda code running on Spark. Due to this only driver node was being used. i did validate this by increasin...

  • 6 kudos
2 More Replies
Labels