How to use CrewAI with Databricks-hosted LLMs (not OpenAI) ?

[lucami]

I am trying to use CrewAI to train an agent with a Databricks-hosted LLM (such as Llama 2 instruct or DBRX) via the databricks_langchain integration. I want CrewAI to use only my Databricks LLM and not requiring OpenAI authentication.

However, when I use ChatDatabricks to both the Agent and the Crew, CrewAI still tries to use OpenAI and fails with an authentication error. If I try to use custom llm, I get litellm.AuthenticationError: databricksException.

• What is the correct way to configure CrewAI to use only a Databricks-hosted LLM ?
• Should I pass the LLM only to the Agent, or also to the Crew?
• Are there any best practices or working examples for this setup in Databricks notebooks?

Thank you!

[lingareddy_Alva]

@lucami

CrewAI defaults to OpenAI unless you explicitly override it.
You must pass your Databricks LLM to both the Agent and the Crew.

Create the Databricks LLM:
Use it in the Agent and in the Crew (this is critical)

[pavannaidu]

@lucami 

To configure CrewAI to use only Databricks-hosted models without OpenAI dependencies

1. Use CrewAI's LLM class (which wraps litellm) rather than ChatDatabricks

2. Pass the LLM at both levels: agents use it to complete their tasks, and the crew uses it to coordinate and orchestrate the overall workflow (as @lingareddy_Alva already suggested) 

3. Enable MLflow tracing for comprehensive execution tracking -- this is super helpful to trace agent execution and model calls.

CrewAI is supported in MLflow!  I tried to replicate the example in MLflow page to use Databricks hosted models > https://gist.github.com/pavannaidu/a61c25fe76b07601bee4aa5138aefffe and see the attached trace for model calls -- hope this helps!