best practice of using data bricks API

[shawncao]

Hello,

I'm building a Databricks connector to allow users to issue command/SQL from a web app.

In general, I think the REST API is okay to work with, though it's pretty tedious to write wrap code for each API call.

[Q1]

Is there an official (or semi-official) client SDK out there I can use? (interested in NodeJS SDK).

[Q2]

Another specific question, if my SQL command produces huge results (> millions of rows), are all results included in one call of "/command/status"? is there a streaming API that we can read row by row instead to reduce the memory consumption?

Thanks for your help!

[Prabakar]

@Shawn Cao​ , Databricks recommends that you use dbx by Databricks Labs for local development instead of Databricks Connect. Databricks plans no new feature development for Databricks Connect at this time.

Please refer to

https://docs.databricks.com/dev-tools/dbx.html

https://github.com/databrickslabs/dbx

[jose_gonzalez]

Hi @Shawn Cao​,

Just a friendly follow-up. Did Prabakar's responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.

[shawncao]

I don't know if I fully understand DBX, sounds like a job client to manage jobs and deployment and I don't see NodeJS support for this project yet.

My question was about how to "stream" query results back from Databricks in a NodeJs application, current solution of using `/command` is not scalable, it can't handle large results.