cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

swzzzsw
by New Contributor III
  • 4241 Views
  • 6 replies
  • 0 kudos

Resolved! SQLServerException: deadlock

I'm using databricks to connect to a SQL managed instance via JDBC. SQL operations I need to perform include DELETE, UPDATE, and simple read and write. Since spark syntax only handles simple read and write, I had to open SQL connection using Scala an...

image.png
  • 4241 Views
  • 6 replies
  • 0 kudos
Latest Reply
Panda
Valued Contributor
  • 0 kudos

@swzzzsw Since you are performing database operations, to reduce the chances of deadlocks, make sure to wrap your SQL operations inside transactions using commit and rollback.Another approachs to consider is adding retry logic or using Isolation Leve...

  • 0 kudos
5 More Replies
iarregui
by New Contributor
  • 2329 Views
  • 2 replies
  • 0 kudos

Getting a Databricks static IP

Hello. I want to connect from my Databricks workspace to an external API to extract some data. The owner of the API asks for an IP to provide the token necessary for the extraction of data. Therefore I would need to set a static IP in Databricks that...

  • 2329 Views
  • 2 replies
  • 0 kudos
Latest Reply
Wojciech_BUK
Valued Contributor III
  • 0 kudos

Hello, the easiest way (in Azure) is to deploy Workspace in VNET Injection mode and attach NAT Gateway to you VNET. NAT GW require Public IP. This IP will be your static egress IP for all Cluster in for this Workspace.Note: both NAT GW and IP Address...

  • 0 kudos
1 More Replies
James1100
by New Contributor II
  • 1561 Views
  • 1 replies
  • 1 kudos

Databricks connect to GCS

Hi,Would like to ask if anyone knows how to connect to GCS - basically read csv file from GCS bucket.I have no issue connecting to Data Lake.Thank you so much in advance.

  • 1561 Views
  • 1 replies
  • 1 kudos
Latest Reply
Vartika
Databricks Employee
  • 1 kudos

Hi @James C​,Just checking in. If @Kaniz Fatma​'s answer helped, would you let us know and mark the answer as best? If not, would you be happy to give us more information?We'd love to hear from you.Cheers!

  • 1 kudos
naveenprabhun
by New Contributor III
  • 4343 Views
  • 2 replies
  • 3 kudos

Resolved! Unable to read data from ElasticSearch using Databricks (AWS) Cannot detect ES version - Caused by: org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed; tried [IP:PORT]

I am trying to read data from ElasticSearch(ES Version 8.5.2) using PySpark on Databricks (13.0 (includes Apache Spark 3.4.0, Scala 2.12)). The ecosystem is on AWS.I am able to run a curl command on the Databricks notebook to the ES ip:port and fetch...

ErrorScreenshot Screenshot 2023-06-01 at 1.25.29 PM
  • 4343 Views
  • 2 replies
  • 3 kudos
Latest Reply
Hoviedo
New Contributor III
  • 3 kudos

I have the same problem, did you find any solution? thanks

  • 3 kudos
1 More Replies
Edwin
by New Contributor II
  • 856 Views
  • 0 replies
  • 1 kudos

Unable to load data from Redshift

I've been trying to connect to RedShift following Databrick's documentation and validated that I'm using runtime version 11.3 on my cluster and that I have read/write privileges on the tempdir bucket. But, I'm unable to load data from RedShift to a S...

  • 856 Views
  • 0 replies
  • 1 kudos
Aj2
by New Contributor III
  • 9789 Views
  • 4 replies
  • 1 kudos

Resolved! How to connect to DB2-AS400?

What are the steps needed to connect to a DB2-AS400 source to pull data to lake using Databricks? I believe it requires establishing a jdbc connection, but I couldnot find much details online

  • 9789 Views
  • 4 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Ajay Menon​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 1 kudos
3 More Replies
rami1
by New Contributor II
  • 7524 Views
  • 2 replies
  • 4 kudos

METASTORE_DOWN: Cannot connect to metastore

I am trying to view databases and tables, default as well user created but it looks like the cluster created is not able to connect. I am using databricks default hive metastore. Viewing cluster logs provide following ventMETASTORE_DOWN Metastore is...

  • 7524 Views
  • 2 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

@rami​ :If the metastore is down, it means that the Databricks cluster is not able to connect to the metastore. Here are a few things you can try to resolve the issue:Check if the Hive metastore is up and running. You can try to connect to the metast...

  • 4 kudos
1 More Replies
youssefmrini
by Databricks Employee
  • 1297 Views
  • 1 replies
  • 0 kudos
  • 1297 Views
  • 1 replies
  • 0 kudos
Latest Reply
youssefmrini
Databricks Employee
  • 0 kudos

Make sure to watch the following video https://www.youtube.com/watch?v=DkzwFTC7WWsThis section lists the requirements for Databricks Connect.Only Databricks Runtime 13.0 ML and Databricks Runtime 13.0 are supported.Only clusters that are compatible w...

  • 0 kudos
Geetha
by New Contributor
  • 2326 Views
  • 2 replies
  • 0 kudos

Connection to Databricks through KNIME

I want to make a connection to Databricks with KNIME. To do this I am using "Create Databricks Environment" node. I have made the following configuration:1. Installed Databricks Simba JDBC driver 2. Made the necessary configuration in Create Databric...

  • 2326 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Geethanjali Nataraj​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell...

  • 0 kudos
1 More Replies
Arnold_Souza
by New Contributor III
  • 6665 Views
  • 4 replies
  • 1 kudos

Connect Databricks to a database protected by a firewall

We a facing a situation and I would like to understand from the Databricks side what is the best practice regarding that. Question: Is it possible to have a cluster with a fixed Global IP on Databricks?DetailsWe have a vendor that has a SQL Server da...

Diagram
  • 6665 Views
  • 4 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

@Arnold Souza​ If you file a support to Azure support they can help customize the Vnet by unlocking it as the Azure Databricks resources are deployed in a managed resource group. Your plan B also should be the way to go if option 1 does not work as e...

  • 1 kudos
3 More Replies
Rishabh-Pandey
by Esteemed Contributor
  • 10212 Views
  • 7 replies
  • 8 kudos

Resolved! connect databricks to teradata

hey i want to know can we connect databricks to the teradata database and if yes what will be the procedure ??? help would be appreciated

  • 10212 Views
  • 7 replies
  • 8 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 8 kudos

use the JDBC driver from here https://docs.databricks.com/integrations/jdbc-odbc-bi.html

  • 8 kudos
6 More Replies
ramravi
by Contributor II
  • 2197 Views
  • 1 replies
  • 0 kudos

Unable to connect to databricks cluster from Windows using databricks-connect

I am trying to setup databricks-connect in my windows machine. While doing databricks-connect test I am getting the below error complaining java certificate is not found. ''Caused by: sun.security.validator.ValidatorException: PKIX path building fail...

cer
  • 2197 Views
  • 1 replies
  • 0 kudos
Latest Reply
ramravi
Contributor II
  • 0 kudos

Adding the certificate from the root level worked for me. This problem is solved.

  • 0 kudos
Sujitha
by Databricks Employee
  • 1535 Views
  • 3 replies
  • 5 kudos

Weekly Release Notes Recap�� Here’s a quick recap of the latest release notes updates from the past one week. Databricks platform release notes...

Weekly Release Notes Recap Here’s a quick recap of the latest release notes updates from the past one week.Databricks platform release notes December 1-6, 2022Partner Connect supports connecting to AtScale:You can now easily create a connection betwe...

  • 1535 Views
  • 3 replies
  • 5 kudos
Latest Reply
karthik_p
Esteemed Contributor
  • 5 kudos

@Uma Maheswara Rao Desula​ if i am not wrong, below ideas portal should help you Ideas Portal | Databricks on AWS

  • 5 kudos
2 More Replies
Labels