topic Query delta tables using databricks cluster in near real time. in Data Engineering

Query delta tables using databricks cluster in near real time.

AmanSehgal — Thu, 10 Feb 2022 08:30:25 GMT

I'm trying to query delta tables using JDBC connector in a Ruby app. I've noticed that it takes around 8 seconds just to connect with databricks cluster and then additional time to run the query.

The app is connected to a web portal where users generate reports. A delay of 10 seconds is a lot and I want to reduce the connection time.

I know I might be able to achieve this using delta sharing, but it's not yet in public preview.

Is there a better way to do this?

Note: I'm using databricks on AWS.

Re: Query delta tables using databricks cluster in near real time.

AmanSehgal — Thu, 10 Feb 2022 14:01:15 GMT

Hi @Kaniz Fatma ,

Great to meet you here on the forum.

I've registered for waitlist on live tables and delta sharing on databricks. Would it be possible to get access to that while a community member responds to my query?

I believe access to it can solve the challenges I'm facing currently.

I'll be more than happy to share my experience with live tables and delta sharing with the community.

Any assistance will be really beneficial for the project.

Re: Query delta tables using databricks cluster in near real time.

AmanSehgal — Fri, 11 Feb 2022 10:29:12 GMT

@Kaniz Fatma can you help me with this issue?

Re: Query delta tables using databricks cluster in near real time.

Anonymous — Tue, 15 Feb 2022 17:00:43 GMT

@Aman Sehgal - We are looking for the best person to help you. Thanks for your patience.

Re: Query delta tables using databricks cluster in near real time.

User16763506477 — Tue, 15 Mar 2022 10:14:48 GMT

Hi @Aman Sehgal Could you please check SQL endpoints? SQL endpoint uses a photon engine. It can reduce the query processing time. And Serverless SQL endpoint can accelerate the launch time

more info: https://docs.databricks.com/sql/admin/sql-endpoints.html

Re: Query delta tables using databricks cluster in near real time.

AmanSehgal — Wed, 16 Mar 2022 22:15:15 GMT

Hi @Gaurav Rupnar

I want to query delta table from a Ruby application. At the moment I'm doing it using SimbaODBC connector and the issue is that it takes 1s to 1.5s to establish connection.

Is there a way that I can do the same via SQL endpoint?