Optimizing Databricks LLM Pipelines with DSPy

lara_rachidi — Fri, 14 Jun 2024 17:25:14 GMT

In October 2023, researchers working in Databricks co-founder Matei Zaharia’s Stanford research lab released DSPy, a library for compiling declarative language model calls into self-improving pipelines. The key component of DSPy is self-improving pipelines. In a complex, multi-stage LLM pipeline, there are often multiple prompts along the way that require tuning. We discuss a blog post that dives into how to build a custom, multi-tool LLM agent using readily available Databricks Marketplace models in DSPy and how to deploy the resulting chain to Databricks Model Serving. Link to blog: https://www.databricks.com/blog/optimizing-databricks-llm-pipelines-dspy

article Optimizing Databricks LLM Pipelines with DSPy in Databricks TV

Optimizing Databricks LLM Pipelines with DSPy