LLM API Returning Inconsistent Responses When Using RAG

[soumitradutta2]

Hi Everyone,

I am Soumitra Dutta,

I have an issue with LLM API and Retrieval-Augmented Generation (RAG). Sometimes the same query returns different answers even if the retrieved documents and the prompt are the same.

The retrieval results are useful, but sometimes the model does not include information from the context that was provided or generates information that does not appear in the source documents.

Have you had any problems with RAG? May it be structure, temperature, token limits or retrieval configuration?

If you have any tips for debugging and making the response be more consistent, it's appreciated.

Regards,
Soumitra Dutta

[balajij8]

@soumitradutta2 

The issue is likely driven by the parameters specifically temperature. Any temperature value above 0 introduces stochastic sampling during token generation that leads to non deterministic outputs even when the prompt and retrieved documents are identical. You can enforce consistency by setting temperature low between 0 and 0.3 if feasible and configuring bounds like max_tokens either directly at inference or inside the model definition using model_config if using ML serving. You can verify the upstream retrieval configuration. Upstream factors like document chunk size, inconsistent chunk formatting, top k selection and similarity threshold cutoffs heavily influence the density and relevance of the context passed to the prompt. If the retrieved chunks are fragmented, inconsistent or border on token limits, the LLM will generally struggle with context adherence and often hallucinate details outside the source data.

You can use tracing to check if the breakdown is happening in retrieval or generation. Trace captures the payload, retrieved document chunks and model outputs for each execution so you can inspect where the variance is. You can also use ML flow Evaluation with RAG judges (Retrieval Relevance, Retrieval Grounded ness and Retrieval Sufficiency) alongside other metrics to objectively score the pipeline. Evaluating the retrieval and LLM generation stages independently against a curated evaluation dataset of expected facts will help you systematically catch hallucinations, context insufficiency and benchmark output stability.