topic Re: Speeding up Command Execution API in Data Engineering

Speeding up Command Execution API

erigaud — Thu, 23 Jan 2025 13:50:16 GMT

Hello,

I'm using the Databricks Command Execution api to run some spark computations on a dedicated cluster and return the results.

I would ideally want the results quickly, especially since the spark computations needed take less that 0.1s.

However when running the command, the status is "Queued" for 5-10s, even though the cluster is not used for anything else.

Is there any way to speed up the command execution ?

Re: Speeding up Command Execution API

Alberto_Umana — Thu, 23 Jan 2025 13:59:42 GMT

Hi @erigaud,

Before running the commands, how have you validated nothing is running / cluster is not in use?

A few suggestions:

Make sure that the cluster is in a running state before you submit the command. The CommandExecutionAPI has a method to ensure the cluster is running

Creating a new execution context for each command can add overhead. Instead, create an execution context once and reuse it for multiple commands. This can be done by creating an execution context and then using it for subsequent commands

// Create an execution context

Created context = commandExecutionService.create(new CreateContext(clusterID, language));

String contextId = context.getId();

// Execute commands using the created context

Command command = new Command(clusterID, contextId, language, commandStr);

Created commandResponse = commandExecutionService.execute(command);

Re: Speeding up Command Execution API

Rjdudley — Thu, 23 Jan 2025 14:46:41 GMT

My first thought concurs with @Alberto_Umana, the only time I've seen queuing like that is when the cluster is not running. Make sure you have the correct warehouse_id configured in your API calls.