Trouble Using Foundation Models API with Instructor

[Gromduraz]

Hello,

I'm attempting to use the instructor library for validating the output of a foundation model (DBRX in this case). Here's my code:

import instructor
from instructor.client import Instructor
from pydantic import BaseModel
from openai import OpenAI

class User(BaseModel):
    name: str
    age: int

client = instructor.from_openai(
    OpenAI(
        api_key=mykey,
        base_url=myendpoint,
    )
)


chat_completion = client.chat.completions.create(
        messages=[
            {
                "role": "system",
                "content": "You are an AI assistant",
            },
            {"role": "user", "content": "Create a user"},
        ],
        model="databricks-dbrx-instruct",
        response_model=User,
        max_tokens=4000,
        temperature=0.8,
        tool_choice="auto",
    )

print(chat_completion.choices[0].message.content)

When I run this I get:

BadRequestError: Error code: 400 - {'error_code': 'BAD_REQUEST', 'message': 'Bad request: json: unknown field "tool_choice"\n'}

I assume the conflict here is that the instructor library was designed to send it's requests directly to OpenAI's servers, while we're using the Databricks Foundation Models API and it's passing along the "tool_choice" field which the endpoint isn't expecting? 

Am I missing something obvious? Can I get this to run as-is or would this require a change to the functionality of the Foundation Models API to run? Are there alternatives to Instructor that I should try? Thanks!

[Kaniz]

Hi @Gromduraz, It appears that you’re encountering an issue with the Instructor library while trying to validate the output of a foundation model (specifically, the DBRX model).

• The error message you received indicates that the field “tool_choice” is unknown in the context of the Databricks Foundation Models API.
• This suggests that the tool_choice parameter is not recognized by the Databricks API, leading to the BadRequestError.
• Remove the “tool_choice” Parameter:
  • Since the Databricks Foundation Models API does not expect the “tool_choice” field, try removing it from your request.
• Check Databricks API Documentation:
  • Review the Databricks Foundation Models API documentation to ensure that you’re using the correct parameters and options.
  • Verify whether there are any specific requirements or limitations related to the model you’re using.
• Alternative Libraries:
  • If the issue persists, consider exploring alternative libraries or approaches for validating LLM outputs.
  • Some other libraries that might be worth trying include:
    • Anthropic: Works with Anthropic models¹.
    • Cohere: Integrates with Cohere models¹.
    • Litellm: Utilizes Litellm completion
•  I hope this helps you troubleshoot the issue, and feel free to explore alternative options if needed...

[Gromduraz]

Reviewing my posted code I should mention that the 'tool_choice' argument I'm passing is a default argument of the OpenAI client object. The code still breaks in the same way if I remove that argument from the function call. I think this is out of my hands to fix as it would require either a change to what instructor sends during the payload to DBX or how DBX interprets the payload. 

I did however find a workaround using LangChain's PydanticValidator. That seemed to work although it's not as direct of an implementation as instructor.