Why keep both Azure OpenAI and Databricks?

[jact]

Hi everyone,

I’m curious to hear your thoughts on the benefits of having both Azure OpenAI and Azure Databricks within the same ecosystem.

From what I can see, Databricks provides a strong foundation for data engineering, governance, and model lifecycle management, while Azure OpenAI offers access to high-performance LLMs through a managed API.

However, I’m trying to better understand why an organization should keep both, instead of consolidating everything inside Databricks (for example, using Databricks Model Serving, vector search, or DBRX).

In which cases does it make sense to:

• Keep Azure OpenAI for inference while using Databricks for data and orchestration?

• Integrate them tightly (e.g., RAG or agentic workflows that use both platforms)?

• Or alternatively, move fully to Databricks' native LLM and serving capabilities?

Would appreciate hearing real-world experiences, architectural best practices, or strategic considerations (e.g., cost, latency, governance, security).

Thanks in advance!

[nayan_wylde]

Two use case I can think of is 

RAG:Use Databricks for vector indexing (e.g., via Delta Lake or FAISS) and Azure OpenAI for inference.
Example: A chatbot that queries Databricks-hosted documents and uses GPT-4 for response generation.

Agentic Workflows:

Combine Databricks orchestration (e.g., MLflow Pipelines, Unity Catalog) with Azure OpenAI for reasoning and decision-making.

My thought is Use both when you need best-in-class LLMs and robust data engineering.
Consolidate into Databricks when governance, cost, and performance are paramount—and when OpenAI models are natively available via Agent Bricks.