Databricks is positioning the platform as a full stack for LLM development — from data ingestion → feature/embedding pipelines → fine-tuning (Mosaic AI) → evaluation → deployment (Model Serving) → monitoring (Lakehouse Monitoring).

I’m curious about real-world experiences here:

1. Where do teams still rely on external components?

• Vector search engines (Pinecone, Weaviate, Milvus) vs. Databricks Vector Search

• Custom LLM gateways (OpenAI, Azure OpenAI, vLLM) vs. Databricks Model Serving

• Feature/embedding stores outside Unity Catalog

• CI/CD + model registry workflows (MLflow vs. SageMaker/Vertex pipelines)

• Guardrails (Guardrails AI, Rebuff, LlamaGuard, Azure Content Filters)

Do you still find gaps in scalability, latency, or retrieval quality?

2. Does Databricks cover everything needed for enterprise-grade evaluation?

• Automated hallucination scoring

• Context relevancy and retrieval precision/recall

• LLM-as-a-judge evaluation pipelines

• Benchmark reproducibility across model versions

• Multi-dataset evaluation (synthetic + real queries)

Or do you still need external tools like TruLens, Ragas, LangSmith, or DeepEval?

3. How mature is drift + quality monitoring for LLMs?

Text-based model drift detection is very different from regression/classification drift:

• Prompt distribution drift

• Embedding drift

• Retrieval degradation over time

• “Silent” quality drops in generative output

• Detecting overfitting after fine-tuning

• Latency/throughput instability for LLM-serving endpoints

Does Lakehouse Monitoring catch these fast enough?