Databricks-Native AI Agent for Job Incident Detection, RCA & Safe Remediation

VibinRoy_C — Sat, 05 Sep 2026 11:29:57 GMT

I’m exploring an architecture for a Databricks-native AI Agent for intelligent Spark job incident detection, root cause analysis (RCA), and safe remediation, and I would love your technical feedback.

The Problem

A common operational challenge is the "Silent Long-Runner" or stalled workload—for example, a Spark job that normally finishes in ~1 hour suddenly runs for 8+ hours with zero meaningful progress, wasting DBUs and blocking pipelines.

Currently, investigating this requires engineers to manually correlate disparate data sources:

Spark UI: Job/Stage DAGs, Task skew, stragglers, shuffle memory/disk spill.
Compute State: Driver/Executor availability, OOMs, memory pressure, thread dumps.
Logs & I/O: Driver/executor logs and real-time input/output progression.
Historical Context: Baseline metrics from past successful runs.

Proposed Concept: Execution-Aware Agentic Intelligence

Instead of a simple log-parsing chatbot, the goal is a closed-loop agent framework built on Databricks-native capabilities (System Tables, MLflow, Agent Framework/MCP, Unity Catalog):

Baseline Learning: Dynamically detects anomalies against historical execution patterns instead of static time limits.
Multi-Signal RCA & Evidence Chain: Correlates live Spark UI metrics, logs, and compute telemetry to yield an explainable diagnosis (e.g., "Root Cause: Executor OOM & Disk Spill Skew — 92% Confidence").
Policy-Governed Remediation: Uses an Observe → Recommend → Auto-Remediate model, requiring Human-in-the-Loop (HITL) approval for high-risk actions (e.g., job cancellation, cluster resizing).
Closed Feedback Loop: Captures whether actions resolved the issue to continuously improve accuracy and quantify DBU/cost savings.

Questions for the Community

Telemetry Access: How much granular Spark UI data (stage progress, task skew, shuffle details) can be programmatically accessed in real time via System Tables or APIs?
Native Pattern: What native capabilities would you recommend for correlating these runtime and log signals efficiently?
Remediation Safety: What guardrails or design patterns do you recommend for automated or HITL actions on running jobs?
Prior Art: Has anyone built or tested a similar agentic incident detection workflow inside Databricks?

Would love to hear your thoughts, feedback, or architectural suggestions!

#genAI #dataengineering #AIagent

topic Databricks-Native AI Agent for Job Incident Detection, RCA & Safe Remediation in Generative AI

Databricks-Native AI Agent for Job Incident Detection, RCA & Safe Remediation

The Problem

Proposed Concept: Execution-Aware Agentic Intelligence

Questions for the Community