Query delta tables using databricks cluster in near real time.

[AmanSehgal]

I'm trying to query delta tables using JDBC connector in a Ruby app. I've noticed that it takes around 8 seconds just to connect with databricks cluster and then additional time to run the query.

The app is connected to a web portal where users generate reports. A delay of 10 seconds is a lot and I want to reduce the connection time.

I know I might be able to achieve this using delta sharing, but it's not yet in public preview.

Is there a better way to do this?

Note: I'm using databricks on AWS.

[Kaniz]

Hi @Aman Sehgal​ ! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will get back to you soon. Thanks.

[AmanSehgal]

Hi @Kaniz Fatma​ ,

Great to meet you here on the forum.

I've registered for waitlist on live tables and delta sharing on databricks. Would it be possible to get access to that while a community member responds to my query?

I believe access to it can solve the challenges I'm facing currently.

I'll be more than happy to share my experience with live tables and delta sharing with the community.

Any assistance will be really beneficial for the project.

[AmanSehgal]

@Kaniz Fatma​  can you help me with this issue?

[Kaniz]

Hi @Aman Sehgal​ , I hope you must have gone through the documentation before requesting access for the live tables and delta sharing on databricks. I shall get back you with an update on your waitlist soon. Thanks.

[Kaniz]

Hi @Aman Sehgal​ , Databricks is excited to announce the general availability of Delta Live Tables.

[Anonymous]

@Aman Sehgal​ - We are looking for the best person to help you. Thanks for your patience.

[User16763506477 · Accepted solution]

Hi @Aman Sehgal​ Could you please check SQL endpoints? SQL endpoint uses a photon engine. It can reduce the query processing time. And Serverless SQL endpoint can accelerate the launch time

more info: https://docs.databricks.com/sql/admin/sql-endpoints.html

[AmanSehgal]

Hi @Gaurav Rupnar​ 

I want to query delta table from a Ruby application. At the moment I'm doing it using SimbaODBC connector and the issue is that it takes 1s to 1.5s to establish connection.

Is there a way that I can do the same via SQL endpoint?

[Kaniz]

Hi @Aman Sehgal​ , Please let me know if this article helps you.