topic Re: best practice of using data bricks API in Data Engineering

best practice of using data bricks API

shawncao — Tue, 26 Apr 2022 20:48:18 GMT

Hello,

I'm building a Databricks connector to allow users to issue command/SQL from a web app.

In general, I think the REST API is okay to work with, though it's pretty tedious to write wrap code for each API call.

[Q1]

Is there an official (or semi-official) client SDK out there I can use? (interested in NodeJS SDK).

[Q2]

Another specific question, if my SQL command produces huge results (> millions of rows), are all results included in one call of "/command/status"? is there a streaming API that we can read row by row instead to reduce the memory consumption?

Thanks for your help!

Re: best practice of using data bricks API

Prabakar — Thu, 02 Jun 2022 06:20:24 GMT

@Shawn Cao , Databricks recommends that you use dbx by Databricks Labs for local development instead of Databricks Connect. Databricks plans no new feature development for Databricks Connect at this time.

Please refer to

https://docs.databricks.com/dev-tools/dbx.html

https://github.com/databrickslabs/dbx

Re: best practice of using data bricks API

jose_gonzalez — Mon, 25 Jul 2022 21:47:02 GMT

Hi @Shawn Cao,

Just a friendly follow-up. Did Prabakar's responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.

Re: best practice of using data bricks API

shawncao — Mon, 25 Jul 2022 22:17:57 GMT

I don't know if I fully understand DBX, sounds like a job client to manage jobs and deployment and I don't see NodeJS support for this project yet.

My question was about how to "stream" query results back from Databricks in a NodeJs application, current solution of using `/command` is not scalable, it can't handle large results.