topic Issue with RAG - Review app - endpoint failed to stream the response in Generative AI

Issue with RAG - Review app - endpoint failed to stream the response

RadhikRajendran — Mon, 23 Jun 2025 14:32:34 GMT

Hi Databricks Support,

I am facing issue for using - RAG review app endpoint url. Error message:
The endpoint failed to stream the response. Retrying without streaming.

The RAG endpoint url was created earlier and it worked with review app, at the time when we used databricks-meta-llama-3-1-70b-instruct. We paused the endpoint for a while and now we need to replace databricks-meta-llama-3-1-70b-instruct to databricks-meta-llama-3-3-70b-instruct. I tried deploying the new version of the model to replace this but the container creation is failing. Could you please in replacing llama3 in endpoint ?

Thanks,

Radhika

Re: Issue with RAG - Review app - endpoint failed to stream the response

MariuszK — Wed, 25 Jun 2025 12:06:02 GMT

Can you share error details? I updated it without an porblme using mlflow.

Re: Issue with RAG - Review app - endpoint failed to stream the response

RadhikRajendran — Wed, 25 Jun 2025 16:08:14 GMT

Below is the example that I am using, not sure if the libraries used here needs update for mlflow.
Similarly for libraries used in config as well may need update accordingly.

genai-cookbook/rag_app_sample_code/A_POC_app/pdf_uc_volume/03_deploy_poc_to_review_app.py at main · databricks/genai-cookbook · GitHub