gateway.create route for open source models

[MichaelO]

Am I able to use gateway.create_route in mlflow for open source LLM models?

I'm aware of the syntax for propietary models like for openAI:

from mlflow import gateway
gateway.create_route(
                              name=OpenAI_embeddings_route_name,
                              route_type="llm/v1/embeddings",
                              model={
                                  "name": "text-embedding-ada-002",
                                  "provider": "openai",
                                  "openai_config": {
                                      "openai_api_key": OpenAI_key,
                                                  }
                                    }
)

Can I use it for open source models without api keys?

Any examples?

[Kaniz_Fatma]

Hi @MichaelO, Certainly! The MLflow AI Gateway provides a way to manage and deploy models, including both proprietary and open source ones. 

Let’s explore how you can create a route for an open source model using the MLflow AI Gateway.

What is the MLflow AI Gateway? The MLflow AI Gateway is an enterprise-grade API gateway that allows organizations to manage their Language Model Models (LLMs) and make them available for experimentation and production. It centralizes governance, credential management, and rate limits for model APIs, ensuring secure and efficient access.

Creating a Route for an Open-Source LLM: You can create a route for an open-source LLM using the create_route function from the mlflow.gateway module. Here’s an example of how to create a route for the Llama2-70B-Chat model using the new support for MosaicML Inference APIs on the AI Gateway:

1. from mlflow import gateway
   
   # Replace 'your key' with your actual MosaicML API key
   mosaicml_api_key = "your key"
   
   gateway.create_route(
       name="completion",
       route_type="llm/v1/completions",
       model={
           "name": "llama2-70b-chat",
           "provider": "mosaicml",
           "mosaicml_config": {
               "mosaicml_api_key": mosaicml_api_key
           }
       }
   )
   

This route allows you to query the Llama2-70B-Chat model for text completions.

Example Use Case: Building a RAG Application Suppose you want to build a Retrieval Augmented Generation (RAG) application that answers gardening questions and provides plant care recommendations. You can leverage hosted models (such as Llama2) and MosaicML inference to experiment cost-effectively. For a detailed example, check out the demo and notebooks provided in the Databricks blog post.

Guardrails and Democratization: The AI Gateway ensures that user behaviour doesn’t abuse or overload the system. By distributing routes, organizations can democratize access to LLMs while maintaining control and security.

Remember that this example demonstrates creating a route for an open source model. You can adapt this approach for other open-source LLMs as well. Happy model deployment! 🚀