LLM API Returning Inconsistent Responses When Using RAG

[soumitradutta2]

Hi Everyone,

I am Soumitra Dutta,

I have an issue with LLM API and Retrieval-Augmented Generation (RAG). Sometimes the same query returns different answers even if the retrieved documents and the prompt are the same.

The retrieval results are useful, but sometimes the model does not include information from the context that was provided or generates information that does not appear in the source documents.

Have you had any problems with RAG? May it be structure, temperature, token limits or retrieval configuration?

If you have any tips for debugging and making the response be more consistent, it's appreciated.

Regards,
Soumitra Dutta

[balajij8]

@soumitradutta2 

The issue is likely driven by the parameters specifically temperature. Any temperature value above 0 introduces stochastic sampling during token generation that leads to non deterministic outputs even when the prompt and retrieved documents are identical. You can enforce consistency by setting temperature low between 0 and 0.3 if feasible and configuring bounds like max_tokens either directly at inference or inside the model definition using model_config if using ML serving. You can verify the upstream retrieval configuration. Upstream factors like document chunk size, inconsistent chunk formatting, top k selection and similarity threshold cutoffs heavily influence the density and relevance of the context passed to the prompt. If the retrieved chunks are fragmented, inconsistent or border on token limits, the LLM will generally struggle with context adherence and often hallucinate details outside the source data.

You can use tracing to check if the breakdown is happening in retrieval or generation. Trace captures the payload, retrieved document chunks and model outputs for each execution so you can inspect where the variance is. You can also use ML flow Evaluation with RAG judges (Retrieval Relevance, Retrieval Grounded ness and Retrieval Sufficiency) alongside other metrics to objectively score the pipeline. Evaluating the retrieval and LLM generation stages independently against a curated evaluation dataset of expected facts will help you systematically catch hallucinations, context insufficiency and benchmark output stability.

[GuyBourne]

This is quite a common problem with RAG systems in general, and it's why monitoring and observability are so important. In terms of the generation step, setting the temperature to zero, as the previous commenter suggested, is a strong approach. However, that will not completely guarantee identical outputs. Now with rag systems, depending on the complexity, there are lots of different things that can go wrong.

The issue that they have is that those errors can then compound. So you've got to look at your retrieval accuracy, and test that in isolation. And with your retrieval accuracy, obviously, you've also got to consider how you've chunked your documents.

If that's not the issue, if we're saying all of that is held constant, then the problem is likely with your gen step.

What I'd suggest doing there is running a series of experiments in MLFlow using various prompts for your gen stap to see which one produces the most consistent answers.

Specifically, I'd follow these steps:

1. Set temperature=0 for deterministic output
2. Strengthen the system prompt with explicit boundaries: "Answer only using provided context. Cite sources. If the answer isn't in the context, say 'I don't have that information.'"
3. Optimise retrieval: reduce chunk size to 500-800 tokens, add metadata filtering, consider hybrid search (vector + BM25)
4. Add reranking or similarity thresholds to filter low-quality retrieved documents
5. Log prompts, retrieved context, and responses for debugging; track consistency metrics with MLflow