Databricks-Native AI Agent for Job Incident Detection, RCA & Safe Remediation

[VibinRoy_C]

I’m exploring an architecture for a Databricks-native AI Agent for intelligent Spark job incident detection, root cause analysis (RCA), and safe remediation, and I would love your technical feedback.

The Problem

A common operational challenge is the "Silent Long-Runner" or stalled workload—for example, a Spark job that normally finishes in ~1 hour suddenly runs for 8+ hours with zero meaningful progress, wasting DBUs and blocking pipelines.

Currently, investigating this requires engineers to manually correlate disparate data sources:

• Spark UI: Job/Stage DAGs, Task skew, stragglers, shuffle memory/disk spill.

• Compute State: Driver/Executor availability, OOMs, memory pressure, thread dumps.

• Logs & I/O: Driver/executor logs and real-time input/output progression.

• Historical Context: Baseline metrics from past successful runs.

Proposed Concept: Execution-Aware Agentic Intelligence

Instead of a simple log-parsing chatbot, the goal is a closed-loop agent framework built on Databricks-native capabilities (System Tables, MLflow, Agent Framework/MCP, Unity Catalog):

1. Baseline Learning: Dynamically detects anomalies against historical execution patterns instead of static time limits.

2. Multi-Signal RCA & Evidence Chain: Correlates live Spark UI metrics, logs, and compute telemetry to yield an explainable diagnosis (e.g., "Root Cause: Executor OOM & Disk Spill Skew — 92% Confidence").

3. Policy-Governed Remediation: Uses an Observe → Recommend → Auto-Remediate model, requiring Human-in-the-Loop (HITL) approval for high-risk actions (e.g., job cancellation, cluster resizing).

4. Closed Feedback Loop: Captures whether actions resolved the issue to continuously improve accuracy and quantify DBU/cost savings.

Questions for the Community

1. Telemetry Access: How much granular Spark UI data (stage progress, task skew, shuffle details) can be programmatically accessed in real time via System Tables or APIs?

2. Native Pattern: What native capabilities would you recommend for correlating these runtime and log signals efficiently?

3. Remediation Safety: What guardrails or design patterns do you recommend for automated or HITL actions on running jobs?

4. Prior Art: Has anyone built or tested a similar agentic incident detection workflow inside Databricks?

Would love to hear your thoughts, feedback, or architectural suggestions!

#genAI #dataengineering #AIagent