cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

best practice of using data bricks API

shawncao
New Contributor II

Hello,

I'm building a Databricks connector to allow users to issue command/SQL from a web app.

In general, I think the REST API is okay to work with, though it's pretty tedious to write wrap code for each API call.

[Q1]

Is there an official (or semi-official) client SDK out there I can use? (interested in NodeJS SDK).

[Q2]

Another specific question, if my SQL command produces huge results (> millions of rows), are all results included in one call of "/command/status"? is there a streaming API that we can read row by row instead to reduce the memory consumption?

Thanks for your help!

3 REPLIES 3

Prabakar
Esteemed Contributor III

@Shawn Cao​ , Databricks recommends that you use dbx by Databricks Labs for local development instead of Databricks Connect. Databricks plans no new feature development for Databricks Connect at this time.

Please refer to

https://docs.databricks.com/dev-tools/dbx.html

https://github.com/databrickslabs/dbx

jose_gonzalez
Moderator
Moderator

Hi @Shawn Cao​,

Just a friendly follow-up. Did Prabakar's responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.

shawncao
New Contributor II

I don't know if I fully understand DBX, sounds like a job client to manage jobs and deployment and I don't see NodeJS support for this project yet.

My question was about how to "stream" query results back from Databricks in a NodeJs application, current solution of using `/command` is not scalable, it can't handle large results.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group