Databricks Community

kulkarnigauri3 · 2 weeks ago

OpsPulse- From Operational Signals to Data-Backed Answers with AIBI Genie

Operations teams usually do not have a shortage of data. They have dashboards, reports, KPIs, spreadsheets, and weekly reviews.

The harder part often starts after someone notices that a metric has moved.

Why is backlog increasing?
Is the issue higher inbound volume or lower processing capacity?
Which vendor is actually deteriorating?
Is an increase meaningful, or just normal variation?

I built OpsPulse for that part of the analytics workflow.

OpsPulse is a Genie-powered operations intelligence app built on Databricks Free Edition. It allows an operations manager or analyst to investigate backlog, SLA attainment, throughput, defect rates, downtime, vendors, sites, and processes using natural-language questions.

My goal was not to build another dashboard or simply put a chat interface on top of a table. I wanted to see how far I could take a governed operational analytics experience where the user can move from:

Detect → Diagnose → Investigate

and where the system also knows when the available data is not sufficient to answer a question.

The use case

Imagine an operations manager opens the application and asks:

What operational issues need attention in the latest available data?

Instead of requiring the manager to open multiple dashboards and compare several KPIs manually, OpsPulse evaluates recent operational performance against an appropriate historical baseline.

It can surface issues such as:

backlog growth
SLA deterioration
declining throughput
increasing defect rates
rising downtime

The user can then continue the investigation naturally.

For example:

Why did East Hub backlog increase after August 18?

In the test data, OpsPulse found that average daily inbound volume had increased to approximately 2,742 units, while average daily processed volume reached only about 2,288 units.

That created an average processing gap of approximately 454 units per day.

At the same time, average daily downtime increased from approximately 226 minutes before August 18 to more than 1,029 minutes after August 18.

This gives the user more than a statement that "backlog increased." It provides the operational context needed to understand what changed.

Who I designed OpsPulse for

I designed the app primarily for:

Operations managers
Business analysts
Data analysts
Supply chain and logistics teams
BI teams supporting operational decision-making

These users may understand their business very well without necessarily wanting to write SQL every time they have a follow-up question.

The value of Genie in this scenario is the ability to keep the investigation moving.

A user can start with a broad question, identify an exception, and immediately ask a more specific question without switching tools or rebuilding an analysis.

Architecture

The application uses a relatively simple architecture, but I spent quite a bit of time on the semantic and analytical layer behind Genie.

Synthetic Operations Data
        ↓
Delta Table
operations_daily
        ↓
Curated KPI View
operations_kpi_view
        ↓
Unity Catalog Metadata
+ Governed KPI Definitions
        ↓
AI/BI Genie Agent
        ↓
Databricks AppKit
        ↓
OpsPulse

The main components are:

Databricks Free Edition
Databricks SQL
Unity Catalog
Delta tables / views
AI/BI Genie
Databricks Apps with AppKit

I used a synthetic operations dataset for this project so that the behavior of the application could be tested against known scenarios.

The dataset contains 1,080 operational records covering 30 days, with combinations of:

4 sites
3 vendors
3 operational processes

The data includes measures such as inbound units, processed units, backlog, SLA-eligible units, SLA-met units, defects, labor hours, downtime, and processing time.

I deliberately introduced several patterns into the data, including an East Hub backlog increase, vendor SLA deterioration, a throughput improvement at another site, and a temporary defect-rate increase.

That made it possible to test whether Genie could actually identify and explain the patterns I knew were present.

Building a governed KPI layer

One thing I wanted to avoid was allowing every question to produce a slightly different KPI calculation.

For example, SLA attainment should not be calculated by simply averaging percentages from individual rows.

For aggregated analysis, I defined SLA attainment as:

Total SLA-met units / Total SLA-eligible units

I used the same approach for other measures:

Throughput

Total processed units / Total labor hours

Defect rate

Total defect units / Total processed units

Net backlog change

Sum of ending backlog minus starting backlog

Processing gap

Inbound units minus processed units

These measures were defined explicitly for Genie so that the same business definitions would be used across different questions.

I also added descriptions and metadata to the Unity Catalog view so fields such as labor_hours, processing_gap_units, and backlog_change_units had clear business meaning.

What users can ask

A few examples that worked well during testing were:

What operational issues need attention in the latest available data?

Why did East Hub backlog increase after August 18?

Which vendor experienced the largest decline in SLA attainment after August 16?

Where did defect rates increase abnormally?

Is East Hub's backlog problem primarily caused by higher inbound volume or weaker processing performance?

These questions demonstrate different parts of the workflow.

Some are broad exception-detection questions. Others are diagnostic questions that require Genie to compare periods, calculate weighted KPIs, or break performance down by site, vendor, and process.

The most useful part of the project was actually the testing

The part I learned the most from was not getting Genie to answer more questions.

It was learning how to improve the quality of the answers and prevent unsupported conclusions.

1. Arbitrary KPI thresholds

During an early test, I asked:

What operational issues need attention?

The first approach introduced thresholds such as SLA below a certain percentage or downtime above a certain number of minutes.

The problem was that I had never defined those thresholds as business rules.

So I changed the analytical approach.

For broad operational questions, OpsPulse now compares the latest seven days against the preceding fourteen-day baseline for the same site, vendor, and process.

This made the analysis much more useful because the application looks for deterioration relative to recent normal performance rather than inventing universal thresholds.

2. Choosing the right baseline matters

I also tested a latest-day comparison against the previous seven days.

That initially looked reasonable, but it created another issue.

If an operational problem had already existed for several days, the recent baseline itself contained part of the deterioration. That could make the latest day look relatively normal and hide the issue.

I changed the logic to compare the most recent seven-day period against the preceding fourteen days.

That simple change significantly improved the exception detection.

It was a good reminder that the quality of AI-assisted analytics still depends heavily on the analytical design behind it.

3. Labor hours are not labor cost

One of my favorite tests was:

What can you tell me about labor cost across the sites?

The dataset contains labor_hours, but it does not contain hourly wages, labor rates, or any monetary cost field.

In an early test, the response started treating higher labor hours as an indication of higher labor cost.

That was not supported by the data.

I updated both the Genie instructions and the Unity Catalog metadata to make the distinction explicit:

Labor hours measure operational effort. They are not a financial cost measure.

After the change, OpsPulse correctly responded that labor cost could not be determined from the available dataset.

That was an important result for me because a useful analytical agent should not only know how to calculate something. It should also know when the data does not support the requested conclusion.

4. "Which site is performing best overall?"

Another interesting test was:

Which site is performing best overall?

Initially, Genie created an implicit ranking using SLA, throughput, defects, backlog, and downtime.

But I had never defined a composite performance score or assigned weights to those KPIs.

There is no analytically defensible reason to assume, for example, that SLA should automatically matter more than defect rate, or that throughput should receive a particular weight.

So I added a guardrail.

Without a governed scoring methodology, OpsPulse now compares the sites across the individual KPIs and explains that the "best" site depends on the business priority.

This was another small change, but an important one from a data governance perspective.

Genie at the core of the application

Genie is not an optional feature inside OpsPulse. It is the primary way the user interacts with the operational data.

The experience is designed around a sequence like this:

1. Detect

What needs attention?

2. Diagnose

Why is East Hub backlog increasing?

3. Investigate

Is inbound volume increasing faster than processing capacity?

The application uses Genie to translate those business questions into queries against the curated operational KPI layer and return the results in a form that an operations user can work with.

Without Genie, the user would lose the natural-language path from an operational question to the underlying data and follow-up analysis.

Building the Databricks App

I used the AppKit Genie template in Databricks Apps and connected the application directly to the OpsPulse Genie Agent.

I customized the application around the three-stage workflow:

Detect

Surface emerging operational exceptions across sites, vendors, and processes.

Diagnose

Compare recent performance with historical baselines and identify the metrics contributing to deterioration.

Investigate

Ask follow-up questions in natural language and explore the supporting operational data with Genie.

I intentionally kept the application itself simple.

Most of the work went into the data model, KPI definitions, business metadata, query patterns, testing, and guardrails behind the interface.

What I learned

This project changed the way I think about conversational analytics.

Connecting an AI interface to data is relatively straightforward.

Making the answers consistently useful is much more interesting.

I found myself spending more time thinking about questions such as:

What does "latest" actually mean?
What comparison period is appropriate?
Should this KPI be averaged or weighted?
Is the system identifying a correlation or claiming a cause?
Is a threshold actually defined by the business?
Is the requested conclusion supported by the available fields?
Should the system rank something when no scoring methodology exists?

Those questions are not really AI questions.

They are analytics, semantic modeling, business logic, and governance questions.

And I think that is where tools like AI/BI Genie become particularly interesting for analysts.

Final thoughts

OpsPulse started as a fairly simple idea: let an operations manager ask questions about operational performance.

By the end of the project, the more interesting challenge became making those answers consistent, explainable, and grounded in the available data.

There is still plenty I would extend in a production version, including governed business targets, more sophisticated anomaly detection, additional historical context, alerting, and integration with real operational datasets.

For this challenge, though, I wanted to focus on one thing:

Can a user move naturally from noticing an operational signal to understanding what may be driving it, without losing the analytical discipline behind the answer?

OpsPulse is my attempt at that.