Databricks-Native AI Agent for Job Incident Detection, RCA & Safe Remediation

[VibinRoy_C]

I’m exploring an architecture for a Databricks-native AI Agent for intelligent Spark job incident detection, root cause analysis (RCA), and safe remediation, and I would love your technical feedback.

The Problem

A common operational challenge is the "Silent Long-Runner" or stalled workload—for example, a Spark job that normally finishes in ~1 hour suddenly runs for 8+ hours with zero meaningful progress, wasting DBUs and blocking pipelines.

Currently, investigating this requires engineers to manually correlate disparate data sources:

• Spark UI: Job/Stage DAGs, Task skew, stragglers, shuffle memory/disk spill.

• Compute State: Driver/Executor availability, OOMs, memory pressure, thread dumps.

• Logs & I/O: Driver/executor logs and real-time input/output progression.

• Historical Context: Baseline metrics from past successful runs.

Proposed Concept: Execution-Aware Agentic Intelligence

Instead of a simple log-parsing chatbot, the goal is a closed-loop agent framework built on Databricks-native capabilities (System Tables, MLflow, Agent Framework/MCP, Unity Catalog):

1. Baseline Learning: Dynamically detects anomalies against historical execution patterns instead of static time limits.

2. Multi-Signal RCA & Evidence Chain: Correlates live Spark UI metrics, logs, and compute telemetry to yield an explainable diagnosis (e.g., "Root Cause: Executor OOM & Disk Spill Skew — 92% Confidence").

3. Policy-Governed Remediation: Uses an Observe → Recommend → Auto-Remediate model, requiring Human-in-the-Loop (HITL) approval for high-risk actions (e.g., job cancellation, cluster resizing).

4. Closed Feedback Loop: Captures whether actions resolved the issue to continuously improve accuracy and quantify DBU/cost savings.

Questions for the Community

1. Telemetry Access: How much granular Spark UI data (stage progress, task skew, shuffle details) can be programmatically accessed in real time via System Tables or APIs?

2. Native Pattern: What native capabilities would you recommend for correlating these runtime and log signals efficiently?

3. Remediation Safety: What guardrails or design patterns do you recommend for automated or HITL actions on running jobs?

4. Prior Art: Has anyone built or tested a similar agentic incident detection workflow inside Databricks?

Would love to hear your thoughts, feedback, or architectural suggestions!

#genAI #dataengineering #AIagent

[kunduruanil]

@VibinRoy_C  Great thought process!

Regarding your second question on Native Patterns:
I highly recommend building an ML model for anomaly detection based on previous runs. You can leverage Databricks MLflow to train a model that predicts pipeline failures using labelled historical data—specifically focusing on signals like data mismatches or missing data events. By training on these specific failure indicators, the agent can proactively catch those "silent long-runners" based on historical log correlations before they stall out completely.

[Satyasai]

Exactly Similar situtation , i have crated my own using Pythopn , VetcorDB and Knowledge graph, you can simulate same or Hire me , i can able to do it for you LOL..

See this Video on my Linked In

[ivanvyd]

@VibinRoy_C  the key limitation is telemetry latency: system.lakeflow.job_run_timeline and job_task_run_timeline are useful for baselines, but records typically arrive within an hour. system.compute.node_timeline adds minute-level metrics for classic compute; neither replaces the full live Spark UI dataset.

For live RCA, combine current run and task state from the Jobs API 2.2 with Spark event logs, compute telemetry, and an application heartbeat such as rows, files, or bytes processed. Runtime alone cannot reliably distinguish a slow job from a stalled one.

A practical flow is:

1. Trigger on duration deviation or a missing progress heartbeat.
2. Collect the run state, historical baselines, node metrics, event logs, and progress signal.
3. Let the agent explain the evidence and recommend an action; use MLflow Tracing to record tool calls and evidence.

“Enable cluster log delivery to cloud storage for long-term log retention.”

Keep diagnosis and execution separate. Give the diagnostic identity CAN VIEW and scoped read access; give an executor only CAN MANAGE RUN on approved jobs. Recheck state before cancel or repair, require approval for high-impact actions, and record the outcome. Cancellation is asynchronous, so an accepted request is not proof the run has stopped.

For a POC, start with duration deviation plus missing progress. Add an ML model only if simpler baselines are insufficient, and publish confidence only after calibration against labeled incidents.