How to reduce query latency?

[Kushal_2612]

Hello everyone,

Context:

I have built an AI agent and deployed it on databricks. Architecture is generally consist of langraph which handles orchestration of whole model, open ai agent which handles response and also use to decide business entity of particular question, databricks mcp tools like execute_sql and ai_search are also used to generate sql query and for handling business questions. Data is stored itself in databricks.

Problem:

I am facing the latency issues in sql_generation part. Data is of 60 years of particular company which is live data so it generally syncs everyday. As an end product I am creating speech to speech so latency should be under 5 seconds on average for all questions. But currently it is coming around 8-10 seconds for small queries. So how can we handle efficiently this to get under 5 seconds?

[juanlozadab · Accepted solution]

I would be careful trying to optimize this only at the SQL level.

For a speech-to-speech experience, 5 seconds is very tight, so I’d first separate the latency by step. In my experience, the delay often comes less from the actual query and more from the chain around it: deciding the entity, retrieving context, generating SQL, validating it, calling the tool, then generating the final answer.

If the user asks small/repeated business questions, I probably wouldn’t generate SQL from scratch every time.

I’d try something like this:

• Define the most common question patterns.
• Create curated gold views or metric tables for those patterns.
• Route simple questions to SQL templates with parameters.
• Use the LLM only to classify the question and extract filters.
• Keep fully dynamic SQL generation only for questions that don’t match a known pattern.

For example, instead of asking the model to generate a full query for “sales last month by region”, the model should only identify:

metric = sales
period = last month
group_by = region

Then your backend builds the SQL deterministically.

That should be much faster and easier to control than letting the agent reason over 60 years of data and a broad schema every time.

I’d also check warehouse cold start, table layout, and pre-aggregations, but architecturally I think the bigger win is avoiding full dynamic SQL generation for the happy path.