cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Query delta tables using databricks cluster in near real time.

AmanSehgal
Honored Contributor III

I'm trying to query delta tables using JDBC connector in a Ruby app. I've noticed that it takes around 8 seconds just to connect with databricks cluster and then additional time to run the query.

The app is connected to a web portal where users generate reports. A delay of 10 seconds is a lot and I want to reduce the connection time.

I know I might be able to achieve this using delta sharing, but it's not yet in public preview.

Is there a better way to do this?

Note: I'm using databricks on AWS.

1 ACCEPTED SOLUTION

Accepted Solutions

User16763506477
Contributor III

Hi @Aman Sehgal​ Could you please check SQL endpoints? SQL endpoint uses a photon engine. It can reduce the query processing time. And Serverless SQL endpoint can accelerate the launch time

more info: https://docs.databricks.com/sql/admin/sql-endpoints.html

View solution in original post

5 REPLIES 5

AmanSehgal
Honored Contributor III

Hi @Kaniz Fatma​ ,

Great to meet you here on the forum.

I've registered for waitlist on live tables and delta sharing on databricks. Would it be possible to get access to that while a community member responds to my query?

I believe access to it can solve the challenges I'm facing currently.

I'll be more than happy to share my experience with live tables and delta sharing with the community.

Any assistance will be really beneficial for the project.

AmanSehgal
Honored Contributor III

@Kaniz Fatma​  can you help me with this issue?

Anonymous
Not applicable

@Aman Sehgal​ - We are looking for the best person to help you. Thanks for your patience.

User16763506477
Contributor III

Hi @Aman Sehgal​ Could you please check SQL endpoints? SQL endpoint uses a photon engine. It can reduce the query processing time. And Serverless SQL endpoint can accelerate the launch time

more info: https://docs.databricks.com/sql/admin/sql-endpoints.html

AmanSehgal
Honored Contributor III

Hi @Gaurav Rupnar​ 

I want to query delta table from a Ruby application. At the moment I'm doing it using SimbaODBC connector and the issue is that it takes 1s to 1.5s to establish connection.

Is there a way that I can do the same via SQL endpoint?

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group