Databricks Serving Endpoint 400 Error: Model Response Format Issue while langgraph tool calling

[kirti-11]

Dear Databricks Community,

I am seeking assistance with an issue I've encountered while deploying a model on Databricks. When invoking the serving endpoint, I receive the following error message sometimes:

400 Client Error: Error: Model response did not respect the required format. Please consider retrying or using a more straightforward prompt.

Context:

• Endpoint URL: https://adb-*****************.11.azuredatabricks.net/serving-endpoints/databricks-meta-llama-3-3-70b...

• Model: databricks-meta-llama-3-3-70b-instruct

• Steps Taken So Far:

  1. Reviewed Input Prompts: Ensured that the input prompts are straightforward and free from complexities that might confuse the model.

  2. Checked Model Response Structure: Verified that the model's output aligns with the expected format required by the serving endpoint.

• Observations:

  • Intermittent Occurrence: The error does not occur consistently; some requests are processed successfully, while others result in the aforementioned error.

• Additional Context:

  • Similar Issues: I have come across similar issues reported by other users, such as the one discussed in this GitHub issue, where the error message 'Model response did not respect the required format' was encountered.

  Despite these efforts, the issue persists intermittently. I would greatly appreciate any insights or recommendations from the community to resolve this error.