Query delta tables using databricks cluster in near real time.

[AmanSehgal]

I'm trying to query delta tables using JDBC connector in a Ruby app. I've noticed that it takes around 8 seconds just to connect with databricks cluster and then additional time to run the query.

The app is connected to a web portal where users generate reports. A delay of 10 seconds is a lot and I want to reduce the connection time.

I know I might be able to achieve this using delta sharing, but it's not yet in public preview.

Is there a better way to do this?

Note: I'm using databricks on AWS.

[AmanSehgal]

Hi @Kaniz Fatma​ ,

Great to meet you here on the forum.

I've registered for waitlist on live tables and delta sharing on databricks. Would it be possible to get access to that while a community member responds to my query?

I believe access to it can solve the challenges I'm facing currently.

I'll be more than happy to share my experience with live tables and delta sharing with the community.

Any assistance will be really beneficial for the project.

[AmanSehgal]

@Kaniz Fatma​  can you help me with this issue?

[Anonymous]

@Aman Sehgal​ - We are looking for the best person to help you. Thanks for your patience.

[User16763506477 · Accepted solution]

Hi @Aman Sehgal​ Could you please check SQL endpoints? SQL endpoint uses a photon engine. It can reduce the query processing time. And Serverless SQL endpoint can accelerate the launch time

more info: https://docs.databricks.com/sql/admin/sql-endpoints.html

[AmanSehgal]

Hi @Gaurav Rupnar​ 

I want to query delta table from a Ruby application. At the moment I'm doing it using SimbaODBC connector and the issue is that it takes 1s to 1.5s to establish connection.

Is there a way that I can do the same via SQL endpoint?