https://community.databricks.com/t5/technical-blog/simplifying-multi-model-llm-development-a-developer-s-guide-to/ba-p/80623 <P><A href="https://docs.litellm.ai/docs/providers/databricks" target="_blank" rel="noopener"><SPAN>LLMs on Databricks are now available to call via LiteLLM</SPAN></A><SPAN>. </SPAN><A href="https://github.com/BerriAI/litellm" target="_blank" rel="noopener"><SPAN>LiteLLM</SPAN></A><SPAN> is a library that provides a python client and an OpenAI-compatible proxy for accessing </SPAN><A href="https://docs.litellm.ai/docs/providers" target="_blank" rel="noopener"><SPAN>100+ LLMs</SPAN></A><SPAN> with the same input/output formats, making it easy to manage and switch between models from different providers. This includes both hosted models (OpenAI/Azure/Bedrock, etc.) and self-hosted models (Ollama/VLLM/TGI, etc.). LiteLLM also works </SPAN><A href="https://docs.litellm.ai/docs/embedding/supported_embedding" target="_blank" rel="noopener"><SPAN>across different endpoints</SPAN></A><SPAN>—chat, completion, embeddings, image generation, etc.</SPAN></P> <H2><STRONG>What Databricks models are available?&nbsp;</STRONG></H2> <P><SPAN>LiteLLM supports models available via the foundation models API, external models, and other chat and embedding models hosted with model serving. This includes chat, completion, and embedding models.</SPAN></P> <P><SPAN>This post will show how to start using LiteLLM with Databricks. We’ll start with a quick discussion of how LiteLLM and Databricks complement each other. Next, we’ll work through a quickstart example of using </SPAN><SPAN>litellm.completion</SPAN><SPAN> to call models from the Databricks Foundation Models API. Next, the post gives a demo of the LiteLLM OpenAI Proxy. We’ll use the proxy to call models from different providers and to log usage. The post concludes with some examples and links to other ways of using LiteLLM with Databricks models.</SPAN></P> <H2><SPAN>Why use LiteLLM with Databricks?</SPAN></H2> <P><SPAN>Using LiteLLM with </SPAN><A href="https://docs.databricks.com/en/machine-learning/model-serving/index.html" target="_blank" rel="noopener"><SPAN>Databricks model serving</SPAN></A><SPAN> builds on the flexibility offered by both for managing and deploying LLMs. Databricks provides robust MLOps capabilities, scalable inference, production-ready observability features, and support for various open-weights models and proprietary models from providers like Anthropic and OpenAI.</SPAN></P> <P><SPAN>LiteLLM complements these capabilities with a unified API for numerous LLM providers and local or self-hosted LLM platforms, simplifying the process of swapping and testing different models or using local models for testing. It also offers additional features such as </SPAN><A href="https://docs.litellm.ai/docs/proxy/cost_tracking" target="_blank" rel="noopener"><SPAN>cost tracking</SPAN></A><SPAN>, </SPAN><A href="https://docs.litellm.ai/docs/proxy/reliability" target="_blank" rel="noopener"><SPAN>error handling</SPAN></A><SPAN>, and </SPAN><A href="https://docs.litellm.ai/docs/proxy/logging" target="_blank" rel="noopener"><SPAN>logging</SPAN></A><SPAN>.</SPAN></P> <P><SPAN>LiteLLM’s support for Databricks models enables developers to:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><A href="https://docs.litellm.ai/docs/#basic-usage" target="_blank" rel="noopener"><SPAN>Quickly prototype</SPAN></A><SPAN> apps across different LLM providers (Databricks/Anthropic/OpenAI/etc.) with a fixed interface</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><A href="https://docs.litellm.ai/docs/routing#advanced---routing-strategies-%EF%B8%8F" target="_blank" rel="noopener"><SPAN>Maximize TPM/RPM limits</SPAN></A><SPAN> across multiple deployments for the same LLM&nbsp;</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><A href="https://docs.litellm.ai/docs/routing#fallbacks" target="_blank" rel="noopener"><SPAN>Implement fallback </SPAN></A><SPAN>mechanisms between Databricks models and other model providers for enhanced reliability</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Transition between self-hosted models and models hosted on Databricks Model Serving for simplified local development and testing.</SPAN></LI> </UL> <H2><SPAN>Quickstart: Using the Python Client</SPAN></H2> <P><SPAN>The LiteLLM Python Client makes it easy to invoke models from different providers via a consistent interface in Python.</SPAN></P> <H3><SPAN>Installation and Setup</SPAN></H3> <P><SPAN>First, install LiteLLM into your Python environment with</SPAN></P> <PRE><SPAN>pip install ‘litellm[proxy]’</SPAN><SPAN>.&nbsp;</SPAN></PRE> <P><SPAN>Next, set up your Databricks model serving credentials:</SPAN></P> <PRE><SPAN>export DATABRICKS_API_KEY=&lt;your_databricks_PAT&gt;</SPAN><BR /><BR /><SPAN>export DATABRICKS_API_BASE=https://&lt;your_databricks_workspace&gt;/serving-endpoints</SPAN></PRE> <P><SPAN>With these setup steps completed, we can start invoking Databricks models using LiteLLM. LiteLLM enables us to use models from any supported provider, including Databricks Model Serving, via the same interface using the LiteLLM Python client.</SPAN></P> <H4><SPAN>Calling DBRX via Databricks Model Serving</SPAN></H4> <P>&nbsp;</P> <LI-CODE lang="python">from litellm import completion response = completion( model="databricks/databricks-dbrx-instruct", messages=[ {"content": "You are a helpful assistant.", "role": "system"}, { "content": "Is this sentence correct? 'Their are many countries in Europe'", "role": "user", }, ], ) print(response)</LI-CODE> <P><SPAN>Which returns:</SPAN></P> <P>&nbsp;</P> <LI-CODE lang="markup">ModelResponse( id='chatcmpl_fee3bc28-562e-4c67-bf14-5628d6cd348c', choices=[ Choices( finish_reason='stop', index=0, message=Message( content='No, the sentence is not correct. The correct form should be "There are many countries in Europe." The word "their" is a possessive pronoun, while "there" is used to indicate a place or to introduce a sentence.', role='assistant' ) ) ], created=1720191284, model='dbrx-instruct-032724', object='chat.completion', system_fingerprint=None, usage=Usage(prompt_tokens=32, completion_tokens=49, total_tokens=81) ) </LI-CODE> <P><SPAN>You can use the same approach to call self-hosted models and models from other providers, simplifying the process of using multiple models in a project without needing to learn each provider’s specific APIs and clients.</SPAN></P> <P><SPAN>Now that you have a basic understanding of why it might be useful to use Databricks model serving with LiteLLM and how to get started with the LiteLLM Python client, let’s look into some of the powerful features enabled via the LiteLLM OpenAI Proxy.</SPAN></P> <H2><SPAN>Demo: Monitoring Usage with the LiteLLM Proxy</SPAN></H2> <P><SPAN>Suppose you are working with a team of developers and want to enable them to access models from multiple providers and keep track of usage. The LiteLLM </SPAN><A href="https://litellm.vercel.app/docs/simple_proxy" target="_blank" rel="noopener"><SPAN>OpenAI Proxy Server</SPAN></A><SPAN> allows us to set up an OpenAI-compatible proxy that lets developers call any supported provider using curl requests or the OpenAI Python SDK. The proxy server includes features such as </SPAN><A href="https://litellm.vercel.app/docs/proxy/virtual_keys" target="_blank" rel="noopener"><SPAN>authentication/key management</SPAN></A><SPAN>, </SPAN><A href="https://litellm.vercel.app/docs/proxy/virtual_keys#advanced---spend-tracking" target="_blank" rel="noopener"><SPAN>spend tracking</SPAN></A><SPAN>, </SPAN><A href="https://litellm.vercel.app/docs/proxy/reliability#quick-start---load-balancing" target="_blank" rel="noopener"><SPAN>load balancing</SPAN></A><SPAN>, and </SPAN><A href="https://litellm.vercel.app/docs/proxy/reliability#quick-start---load-balancing" target="_blank" rel="noopener"><SPAN>fallbacks</SPAN></A><SPAN>.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="litellm_excal.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/9963iDA911C0FDB0027AE/image-size/large?v=v2&amp;px=999" role="button" title="litellm_excal.png" alt="litellm_excal.png" /></span></P> <P><SPAN>In this example, we will use the proxy to give developers access to the Databricks DBRX model and the Anthropic Claude 3.5 Sonnet model, and then log their respective token usages.</SPAN></P> <H3><SPAN>Configure and Start the Proxy Server</SPAN></H3> <P><SPAN>We’ll configure the proxy server with the following </SPAN><SPAN>config.yaml</SPAN><SPAN> file. This configuration will expose the Databricks DBRX model from Databricks Model Serving and Claude 3.5 Sonnet via the Anthropic API.</SPAN></P> <P>&nbsp;</P> <LI-CODE lang="markup">model_list: - model_name: dbrx litellm_params: model: databricks/databricks-dbrx-instruct api_key: os.environ/DATABRICKS_API_KEY api_base: os.environ/DATABRICKS_API_BASE - model_name: claude litellm_params: model: databricks/claude-3-5-sonnet api_key: os.environ/ANTHROPIC_API_KEY </LI-CODE> <P><SPAN>We can then start the server with:</SPAN></P> <PRE><SPAN>litellm --config config.yaml</SPAN></PRE> <H3><SPAN>Call Models with the OpenAI Client</SPAN></H3> <P><SPAN>And call either of these models with OpenAI-compatible methods. For example, we can now use the OpenAI Python client to call DBRX via the LiteLLM proxy as follows:</SPAN></P> <P>&nbsp;</P> <LI-CODE lang="python">import openai client = openai.OpenAI( api_key="anything", base_url="http://0.0.0.0:4000" ) response = client.chat.completions.create(model="dbrx", messages = [ { "role": "user", "content": "Is this sentence correct? 'Their are many countries in Europe'" } ], max_tokens=25)</LI-CODE> <P><SPAN>To use Claude instead, all we need to do is change the model name:</SPAN></P> <PRE><SPAN>response</SPAN> <SPAN>=</SPAN> <SPAN>client.chat.completions.create(model=</SPAN><SPAN>"claude"</SPAN><SPAN>,</SPAN> <SPAN>messages</SPAN> <SPAN>= ...</SPAN></PRE> <P><SPAN>Everything else stays the same, making it very easy to switch between models. You can also use OpenAI-compatible REST API calls via, for example, </SPAN><SPAN>curl</SPAN><SPAN> or the Python </SPAN><SPAN>requests</SPAN><SPAN> library.</SPAN></P> <H3><SPAN>Monitoring Costs and Usage</SPAN></H3> <P><SPAN>The LiteLLM OpenAI Proxy Server lets us </SPAN><A href="https://docs.litellm.ai/docs/proxy/streaming_logging" target="_blank" rel="noopener"><SPAN>log token usage</SPAN></A><SPAN>. This lets us collect cost and usage details in one place, saving us from needing to check the usage dashboards of multiple different LLM providers. We can do this by </SPAN><A href="https://docs.litellm.ai/docs/proxy/streaming_logging#step-1---create-your-custom-litellm-callback-class" target="_blank" rel="noopener"><SPAN>implementing a callback</SPAN></A><SPAN> and adding it to our config. To log usage, create a new file called </SPAN><STRONG><FONT face="andale mono,times">custom_callbacks.py</FONT></STRONG><SPAN> and subclass the </SPAN><STRONG><FONT face="andale mono,times">litellm.integrations.custom_logger.CustomLogger</FONT></STRONG><SPAN> class:</SPAN></P> <P>&nbsp;</P> <LI-CODE lang="python">from litellm.integrations.custom_logger import CustomLogger import litellm import logging class MyCustomHandler(CustomLogger): async def async_log_success_event(self, kwargs, response_obj, start_time, end_time): try: # init logging config logging.basicConfig( filename='cost.log', level=logging.INFO, format='%(asctime)s - %(message)s', datefmt='%Y-%m-%d %H:%M:%S' ) response_cost = kwargs.get("response_cost") input_tokens = response_obj.usage.prompt_tokens output_tokens = response_obj.usage.completion_tokens print("input_tokens", input_tokens, "output_tokens", output_tokens) logging.info(f"Model: {response_obj.model} Input Tokens: {input_tokens} Output Tokens: {output_tokens} Response Cost: {response_cost}") except Exception as e: print(f"Failed to log usage: {e}") proxy_handler_instance = MyCustomHandler()</LI-CODE> <P><SPAN>We also need to update the </SPAN><SPAN>config.yaml</SPAN><SPAN> file to add the cost per input/output token for DBRX. You can use </SPAN><A href="https://www.databricks.com/product/pricing/foundation-model-serving" target="_blank" rel="noopener"><SPAN>this calculator</SPAN></A><SPAN> to determine costs depending on your cloud provider and region. We update the DBRX config entry as follows:</SPAN></P> <P>&nbsp;</P> <LI-CODE lang="markup"> - model_name: dbrx litellm_params: model: databricks/databricks-dbrx-instruct api_key: os.environ/DATABRICKS_API_KEY api_base: os.environ/DATABRICKS_API_BASE input_cost_per_token: 0.00000075 output_cost_per_token: 0.00000225</LI-CODE> <P><SPAN>When we call either model via the OpenAI proxy, the token usage and cost information will be recorded in the usage.log file. If needed, we can aggregate and analyze the log data to get a unified view of usage and costs across providers.</SPAN></P> <P>&nbsp;</P> <LI-CODE lang="markup">2024-07-16 12:17:55 - Model: dbrx-instruct-032724 Input Tokens: 237 Output Tokens: 47 Response Cost: 0.0002835 2024-07-16 12:18:06 - Model: claude-3-5-sonnet-20240620 Input Tokens: 21 Output Tokens: 145 Response Cost: 0.002238 2024-07-16 12:18:28 - Model: claude-3-5-sonnet-20240620 Input Tokens: 23 Output Tokens: 284 Response Cost: 0.0043289 2024-07-16 12:18:32 - Model: dbrx-instruct-032724 Input Tokens: 238 Output Tokens: 81 Response Cost: 0.00036075 2024-07-16 12:18:55 - Model: dbrx-instruct-032724 Input Tokens: 230 Output Tokens: 37 Response Cost: 0.00025575 2024-07-16 12:22:39 - Model: dbrx-instruct-032724 Input Tokens: 230 Output Tokens: 45 Response Cost: 0.00027375 2024-07-16 12:22:54 - Model: claude-3-5-sonnet-20240620 Input Tokens: 15 Output Tokens: 123 Response Cost: 0.00189 2024-07-16 12:23:28 - Model: claude-3-5-sonnet-20240620 Input Tokens: 18 Output Tokens: 195 Response Cost: 0.002979 2024-07-16 12:23:40 - Model: dbrx-instruct-032724 Input Tokens: 234 Output Tokens: 48 Response Cost: 0.0002835</LI-CODE> <P><SPAN>For more advanced user- and team-level monitoring, access management, and spend tracking, you can </SPAN><A href="https://docs.litellm.ai/docs/proxy/virtual_keys" target="_blank" rel="noopener"><SPAN>set up a Postgres database and create API keys</SPAN></A><SPAN>. There is also a </SPAN><A href="https://docs.litellm.ai/docs/proxy/self_serve" target="_blank" rel="noopener"><SPAN>UI</SPAN></A><SPAN> for adding users, creating keys, monitoring usage, and more.</SPAN></P> <H2><SPAN>Other model types</SPAN></H2> <P><SPAN>The examples above focused on completions, but it’s worth noting that LiteLLM supports other types of models as well. For example, we can call the </SPAN><STRONG><FONT face="andale mono,times">gte-large-en</FONT></STRONG><SPAN> embedding model available via the Databricks foundation models API via </SPAN><STRONG><FONT face="andale mono,times">litellm.embeddings</FONT></STRONG><SPAN>:</SPAN></P> <P>&nbsp;</P> <LI-CODE lang="python">from litellm import embedding response = embedding( model="databricks/databricks-gte-large-en", input=["General text embeddings (GTE) can map any text to a low-dimensional dense vector which can be used for tasks like retrieval, classification, clustering, or semantic search. And it also can be used in vector databases for LLMs."], instruction="Represent this sentence for searching relevant passages:", ) print(response)</LI-CODE> <P>&nbsp;</P> <P><SPAN>Which returns:</SPAN></P> <P>&nbsp;</P> <LI-CODE lang="markup">EmbeddingResponse( model='gte-large-en-v1.5', data=[ { 'index': 0, 'object': 'embedding', 'embedding': [ 1.0078125, -0.25537109375, -0.755859375, -0.0692138671875, [...] 1.36328125, -0.2440185546875, -0.2159423828125 ] } ], object='list', usage=Usage(prompt_tokens=62, total_tokens=62) )</LI-CODE> <H2><SPAN>Next Steps</SPAN></H2> <P><SPAN>This was a quick introduction to using Databricks model serving with LiteLLM. After reading this guide, you should be able to:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>Call Databricks Model Serving models, and models from other providers, with LiteLLM</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Set up the LiteLLM OpenAI Proxy Server and call models from different sources with the OpenAI Python Client</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Use Completions, Chat, and Embedding models with LiteLLM</SPAN></LI> </UL> <P><SPAN>There is much more you can do with the combination of Databricks model serving and LiteLLM. Here are some ideas:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>Serve </SPAN><A href="https://docs.databricks.com/en/machine-learning/model-serving/model-serving-intro.html" target="_blank" rel="noopener"><SPAN>custom chat models</SPAN></A><SPAN> with Databricks model serving and add them to your LiteLLM OpenAI Proxy Server</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Add models from other providers: maybe in addition to Databricks and Anthropic models you also want to try out models from some </SPAN><A href="https://litellm.vercel.app/docs/providers" target="_blank" rel="noopener"><SPAN>other providers</SPAN></A><SPAN> or even locally-hosted models via Ollama. You can do so by adding them to your proxy config file or calling them directly from the LiteLLM Python client.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Configure </SPAN><A href="https://litellm.vercel.app/docs/proxy/reliability" target="_blank" rel="noopener"><SPAN>load balancing, fallbacks, retries, and timeouts</SPAN></A><SPAN> with the LiteLLM OpenAI proxy server. This is particularly useful if you expect a high volume of requests and want to ensure reliable performance.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1">Write a <A style="font-family: inherit; background-color: #ffffff;" href="https://litellm.vercel.app/docs/proxy/logging#custom-callback-class-async" target="_blank" rel="noopener">custom callback</A><SPAN> to log model calls/responses to MLflow.</SPAN></LI> </UL> Mon, 05 Aug 2024 16:56:26 GMT Daniel-Liden 2024-08-05T16:56:26Z https://community.databricks.com/t5/technical-blog/simplifying-multi-model-llm-development-a-developer-s-guide-to/ba-p/80623 <DIV class="font-claude-message pr-4 md:pr-9 relative leading-[1.65rem] [&amp;_pre&gt;div]:bg-bg-300 [&amp;_pre]:-mr-4 md:[&amp;_pre]:-mr-9"> <DIV> <DIV class="grid-col-1 grid gap-2.5 [&amp;_&gt;_*]:min-w-0"> <P class="whitespace-pre-wrap break-words">This article introduces the integration of LiteLLM with Databricks, allowing users to access and manage various large language models (LLMs) through a unified interface. It covers how to use LiteLLM's Python client and OpenAI-compatible proxy to call Databricks models, as well as models from other providers. The piece also demonstrates setting up usage monitoring and cost tracking across different LLMs. Examples are provided for using completion, chat, and embedding models via LiteLLM with Databricks model serving.</P> </DIV> </DIV> </DIV> Mon, 05 Aug 2024 16:56:26 GMT https://community.databricks.com/t5/technical-blog/simplifying-multi-model-llm-development-a-developer-s-guide-to/ba-p/80623 Daniel-Liden 2024-08-05T16:56:26Z https://community.databricks.com/t5/technical-blog/simplifying-multi-model-llm-development-a-developer-s-guide-to/bc-p/101476#M406 <P><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/109683">@Daniel-Liden</a>&nbsp;Thanks for this great post - I appreciate the concise overview. It's been a few months now - is this still a recommended way to proxy LLM calls? And if so, where do you recommend serving the LiteLLM proxy?</P><P>If it helps, my use case is considering LangChain and I'm particularly trying to understand whether the proxy would be hosted on the same cluster running the LangChain code (potentially a hosted instance in Model Serving) or a standalone service on a separate cluster.</P> Mon, 09 Dec 2024 14:07:30 GMT https://community.databricks.com/t5/technical-blog/simplifying-multi-model-llm-development-a-developer-s-guide-to/bc-p/101476#M406 pemidexx 2024-12-09T14:07:30Z https://community.databricks.com/t5/technical-blog/simplifying-multi-model-llm-development-a-developer-s-guide-to/bc-p/101478#M407 <P><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/120181">@pemidexx</a>&nbsp;Thanks!</P> <P>I would check out <A href="https://docs.databricks.com/en/generative-ai/external-models/index.html" target="_self">External Models</A> and&nbsp;<A href="https://docs.databricks.com/en/ai-gateway/index.html" target="_self">Mosaic AI Gateway</A> as built-in options.<BR /><BR />External models provides:</P> <UL> <LI>Built-in support for major LLM providers (OpenAI, Anthropic, Cohere, etc.)</LI> <LI>Centralized credential management</LI> <LI>Unified endpoint interface</LI> <LI>Native integration with the rest of the Databricks platform</LI> </UL> <P>So you wouldn't need to run and maintain a separate proxy service.</P> Mon, 09 Dec 2024 14:36:31 GMT https://community.databricks.com/t5/technical-blog/simplifying-multi-model-llm-development-a-developer-s-guide-to/bc-p/101478#M407 Daniel-Liden 2024-12-09T14:36:31Z