cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Speeding up Command Execution API

erigaud
Honored Contributor

Hello,

I'm using the Databricks Command Execution api to run some spark computations on a dedicated cluster and return the results. 

I would ideally want the results quickly, especially since the spark computations needed take less that 0.1s.

However when running the command, the status is "Queued" for 5-10s, even though the cluster is not used for anything else.

Is there any way to speed up the command execution ? 

erigaud_0-1737640182808.png

 

2 REPLIES 2

Alberto_Umana
Databricks Employee
Databricks Employee

Hi @erigaud,

Before running the commands, how have you validated nothing is running / cluster is not in use?

A few suggestions:

Make sure that the cluster is in a running state before you submit the command. The CommandExecutionAPI has a method to ensure the cluster is running

Creating a new execution context for each command can add overhead. Instead, create an execution context once and reuse it for multiple commands. This can be done by creating an execution context and then using it for subsequent commands

// Create an execution context

Created context = commandExecutionService.create(new CreateContext(clusterID, language));

String contextId = context.getId();

// Execute commands using the created context

Command command = new Command(clusterID, contextId, language, commandStr);

Created commandResponse = commandExecutionService.execute(command);

Rjdudley
Valued Contributor II

My first thought concurs with @Alberto_Umana, the only time I've seen queuing like that is when the cluster is not running.  Make sure you have the correct warehouse_id configured in your API calls.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group