gateway.create route for open source models

MichaelO
New Contributor III

Am I able to use gateway.create_route in mlflow for open source LLM models?

I'm aware of the syntax for propietary models like for openAI:

 

from mlflow import gateway
gateway.create_route(
                              name=OpenAI_embeddings_route_name,
                              route_type="llm/v1/embeddings",
                              model={
                                  "name": "text-embedding-ada-002",
                                  "provider": "openai",
                                  "openai_config": {
                                      "openai_api_key": OpenAI_key,
                                                  }
                                    }
)

 

Can I use it for open source models without api keys?

Any examples?