No time to read? Check out this 5-minute recap video of all the announcements listed below for November 2024 👇
→ Subscribe to our YouTube channel for regular updates
Announcements
Mosaic AI Model Training rebrand
Mosaic AI Model Training has been rebranded to encompass existing features:
- Foundation Model Fine-tuning, previously known as Mosaic AI Model Training, continues to offer advanced customization for foundation models.
- AutoML features can now be accessed from the Experiments section.
Other Platform updates
- Foundation Model APIs pay-per-token workloads are now supported in all regions where Mosaic AI Model Serving is available. If your workspace is in a Model Serving region but not in a U.S. or EU region, your workspace must be enabled for cross-Geo data processing. When enabled, your pay-per-token workload is routed to the U.S. Databricks Geo.
- Breaking change: Hosted RStudio is end-of-life. In Databricks Runtime 16.0 and above, Databricks-hosted RStudio Server is end-of-life and unavailable on any Databricks workspace. To learn more and see a list of alternatives to RStudio, see Hosted RStudio Server deprecation.
Demos
- Serving Vision Language Models on Databricks | Mlflow Extensions → Demo
- Using GenAI and Traditional ML for Anomaly & Outlier Detection → Demo
- Optimise RAG applications with semantic caching on Databricks → Demo
Blogposts
Demo: https://www.youtube.com/watch?v=mIZHRqMoJec
Introducing Structured Outputs for Batch and Agent Workflows
- This blog post explores in detail how to use structured outputs in Databricks for batch and agent workflows, along with code snippets.
Use case 1 — Batch structured generation
- For batch structured generation, you can use the
response_formatAPI field to reliably structure JSON outputs from large datasets. This is available for the Foundation Model API models, including fine-tuned models. The three differentresponse_formatare: Text:Unstructured text outputted from the model based on a prompt.Json_object:Output a JSON object of an unspecified schema that the model intuits from the promptJson_schema:Output a JSON object adherent to a JSON schema applied to the API.- Note that the Open AI SDK makes it easy to define object schemas using Pydantic that you can pass to the model instead of an articulated JSON schema.
from pydantic import BaseModel
from openai import OpenAI
DATABRICKS_TOKEN = os.environ.get('YOUR_DATABRICKS_TOKEN')
DATABRICKS_BASE_URL = os.environ.get('YOUR_DATABRICKS_BASE_URL')
client = OpenAI(
api_key=DATABRICKS_TOKEN,
base_url=DATABRICKS_BASE_URL
)
class CalendarEvent(BaseModel😞
name: str
date: str
participants: list[str]
completion = client.beta.chat.completions.parse(
model="databricks-meta-llama-3-1-70b-instruct",
messages=[
{"role": "system", "content": "Extract the event information."},
{"role": "user", "content": "Alice and Bob are going to a science fair on Friday."},
],
response_format=CalendarEvent,
)
print(completion.choices[0].message.parsed)
#name='science fair' date='Friday' participants=['Alice', 'Bob']
Use case 2 — Agents with function calling
- The second use case for structured outputs is when you build Agents with function calling. Function calling allows LLMs to consistently output function calls to APIs within agent workflows. They’re currently available for Llama 3 70B and Llama 405B.
- With the
toolsparameter, you can specify a list of potential tools that the LLM can call, where each tool is a function defined with a name, description, and parameters in the form of a JSON schema. You can then usetool_choiceto determine how tools are called. The options are: none: The model will not call any tool listed in tools.auto: The model will decide the relevance of whether a tool from the tools list should be called or not. If no tool is called, the model outputs unstructured text like normal.required: The model will definitely output one of the tools in the list of tools no matter the relevance{"type": "function", "function": {"name": "my_function"}}: If ”my_function” is the name of a valid function in the list of tools, the model will be forced to pick that function.
from openai import OpenAI
DATABRICKS_TOKEN = os.environ.get('YOUR_DATABRICKS_TOKEN')
DATABRICKS_BASE_URL = os.environ.get('YOUR_DATABRICKS_BASE_URL')
client = OpenAI(
api_key=DATABRICKS_TOKEN,
base_url=DATABRICKS_BASE_URL
)
tools = [
{
"type": "function",
"function": {
"name": "get_delivery_date",
"description": "Get the delivery date for a customer's order. Call this whenever you need to know the delivery date, for example when a customer asks 'Where is my package'",
"parameters": {
"type": "object",
"properties": {
"order_id": {
"type": "string",
"description": "The customer's order ID.",
},
},
"required": ["order_id"],
},
}
},
{
"type": "function",
"function": {
"name": "get_relevant_products",
"description": "Return a list of relevant products that are being sold for a given search query. For example, call this if a customer asks 'What laptops do you have for sale?'",
"parameters": {
"type": "object",
"properties": {
"search_query": {
"type": "string",
"description": "The category of products to search for.",
},
"number_of_items": {
"type": "integer",
"description": "The number of items to return in the search response. Default is 5 and maximum is 20.",
},
},
"required": ["search_query"],
},
}
}
]
response = client.chat.completions.create(
model="databricks-meta-llama-3-1-70b-instruct",
messages=[
{"role": "user", "content": "Do you have any keyboards for sale?"}],
tools=tools,
tool_choice="auto",
)
print(response.choices[0].message.tool_calls)
- They also give tips to use simpler JSON schemas to produce higher quality outputs compared to more complex JSON schemas, and what you should avoid (eg highly nested JSON schemas). They run through an example of what good looks like.
- Link to blog post
AI Agent Systems: Modular Engineering for Reliable Enterprise AI Applications
- This blog post explores the evolution of technological systems from monolithic designs to modular architectures, highlighting the benefits of modularity in managing complexity, enhancing maintainability, and improving extensibility. It traces this trend across industries such as automotive, software, and AI. The transition is particularly relevant to LLMs, which initially operated as monolithic systems with limited flexibility and reliability for enterprise use. Modular AI agent systems address these limitations by separating functions like data retrieval, deterministic processing, and reasoning into independently verifiable components. This enables greater control, compliance with regulatory frameworks, and adaptability for various applications. For instance, healthcare applications can isolate data retrieval from interpretation to meet compliance standards while maintaining system reliability and accuracy.
- Databricks adopted this modular approach with its Mosaic AI framework, which integrates tools for evaluation, monitoring, and governance. These modular AI agent systems combine components like embedding models, vector databases, and fine-tuned LLMs to deliver high-quality outputs. This architecture also allows for the customization of applications to specific domains. The blog concludes that moving from monolithic LLMs to modular intelligence systems represents a paradigm shift in AI application development, enabling higher reliability and extensibility.
- Link to blog post
Securing the Future: How AI Gateways Protect AI Agent Systems in the Era of Generative AI
- Organizations need to go beyond traditional API policies to secure AI agent systems. The key lies in building AI gateways that protect the API layers and evaluate the instructions sent to the AI agent system. This blog explores how AI Gateways can secure AI agent systems, ensuring their safe deployment and operation in today’s complex digital landscape.
- Link to blog post
