Lakeflow Jobs give teams a strong operational surface for individual workflows, but platform teams often need a different view: Which jobs are missing their reliability target? Which pipelines are getting slower? Which failures are concentrated in one owner, workspace, task, or schedule? Which successful jobs are becoming expensive?
System tables make it possible to answer these questions at account scope. The system.lakeflow tables record job and task activity, while the billing tables let you connect operational behavior to cost for jobs compute and serverless jobs.
This post walks through a simple service-level objective, or SLO, dashboard pattern for Lakeflow Jobs. The goal is not to replace the Jobs UI. The goal is to create a shared operational view that platform teams and data product owners can review every week.
Most data teams already have informal expectations:
- The daily finance load should finish before 7:00 AM.
- The customer 360 pipeline should succeed at least 99 percent of the time.
- A warehouse refresh should not suddenly cost twice as much.
- Critical jobs should have an owner and a run-as identity that is not a personal user.
These expectations often live in chat threads, runbooks, or tribal knowledge. That makes them hard to audit and hard to improve.
An SLO dashboard gives you a practical middle ground. It does not need to be perfect on day one. It needs to show the jobs that deserve attention, explain why they are flagged, and give owners enough context to act.
What system tables give you
The Lakeflow jobs system tables are available under system.lakeflow. They include account-level records for jobs, tasks, job runs, task runs, pipelines, and pipeline updates in the current cloud region.
For this pattern, the most useful tables are:
- system.lakeflow.jobs: job definitions and metadata.
- system.lakeflow.job_tasks: job task definitions.
- system.lakeflow.job_run_timeline: job run history and state.
- system.lakeflow.job_task_run_timeline: task run history and state.
- system.billing.usage: billable usage records.
- system.billing.list_prices: pricing records used to estimate list cost.
There are two important boundaries:
- System table access requires the right account and schema permissions.
- Job records are regional. To monitor workspaces in another cloud region, run the queries from a workspace in that region.
Define the first SLOs
Start with a small set of SLOs that a platform team can explain:
SLO Why it matters Example threshold
| Success rate | Reliability of scheduled work | 98 percent success over 30 days |
| P95 duration | Detects jobs that are getting slower | Less than 1.5x the agreed target |
| Freshness | Detects jobs that stopped running | Last successful run within expected schedule |
| Cost per successful run | Finds inefficient or runaway jobs | Within expected range for the job class |
| Ownership coverage | Ensures accountability | Owner tag and non-personal run-as for critical jobs |
You can store formal thresholds in a small reference table. That keeps the dashboard logic clean and lets each domain tune its own targets.
CREATE TABLE IF NOT EXISTS ops_reference.job_slo_targets (
workspace_id STRING,
job_id STRING,
owner_team STRING,
criticality STRING,
min_success_rate_30d DOUBLE,
max_p95_duration_minutes DOUBLE,
max_hours_since_success DOUBLE,
max_cost_per_successful_run DOUBLE
);
For an initial rollout, populate only the most important production jobs. Jobs without a target can still appear in discovery panels, but they should not create false SLO breaches.
Build the operational view
The following view summarizes 30-day job reliability and duration. Adjust the catalog and schema names to match your workspace.
CREATE OR REPLACE VIEW ops_observability.lakeflow_job_slo_30d AS
WITH latest_jobs AS (
SELECT
workspace_id,
job_id,
name AS job_name,
creator_id,
run_as,
tags,
ROW_NUMBER() OVER (
PARTITION BY workspace_id, job_id
ORDER BY change_time DESC
) AS rn
FROM system.lakeflow.jobs
QUALIFY rn = 1
),
runs AS (
SELECT
workspace_id,
job_id,
run_id,
period_start_time,
period_end_time,
result_state,
(unix_timestamp(period_end_time) - unix_timestamp(period_start_time)) / 60.0
AS duration_minutes
FROM system.lakeflow.job_run_timeline
WHERE period_start_time >= current_timestamp() - INTERVAL 30 DAYS
AND result_state IS NOT NULL
),
run_summary AS (
SELECT
workspace_id,
job_id,
count(*) AS total_runs_30d,
count_if(result_state = 'SUCCEEDED') AS successful_runs_30d,
count_if(result_state <> 'SUCCEEDED') AS failed_runs_30d,
count_if(result_state = 'SUCCEEDED') / nullif(count(*), 0) AS success_rate_30d,
percentile_approx(duration_minutes, 0.95) AS p95_duration_minutes,
max(CASE WHEN result_state = 'SUCCEEDED' THEN period_end_time END) AS last_success_time
FROM runs
GROUP BY workspace_id, job_id
)
SELECT
j.workspace_id,
j.job_id,
j.job_name,
j.run_as,
j.creator_id,
j.tags,
s.total_runs_30d,
s.successful_runs_30d,
s.failed_runs_30d,
s.success_rate_30d,
s.p95_duration_minutes,
s.last_success_time,
timestampdiff(HOUR, s.last_success_time, current_timestamp()) AS hours_since_success
FROM latest_jobs j
LEFT JOIN run_summary s
ON j.workspace_id = s.workspace_id
AND j.job_id = s.job_id;This view is intentionally simple. It gives you a stable base for dashboards and alerts before adding task-level detail.
Add cost context
Cost is most useful when it is shown beside operational behavior. A job that fails often and costs little may need code cleanup. A job that succeeds but doubles its cost every week may need a compute or data layout review.
The query below estimates list cost for jobs usage over the last 30 days and joins it to the SLO view.
CREATE OR REPLACE VIEW ops_observability.lakeflow_job_slo_30d_with_cost AS
WITH job_usage AS (
SELECT
u.workspace_id,
u.usage_metadata.job_id AS job_id,
sum(u.usage_quantity * p.pricing.default) AS list_cost_30d
FROM system.billing.usage u
INNER JOIN system.billing.list_prices p
ON u.cloud = p.cloud
AND u.sku_name = p.sku_name
AND u.usage_start_time >= p.price_start_time
AND (u.usage_end_time <= p.price_end_time OR p.price_end_time IS NULL)
WHERE u.billing_origin_product = 'JOBS'
AND u.usage_date >= current_date() - INTERVAL 30 DAYS
GROUP BY u.workspace_id, u.usage_metadata.job_id
)
SELECT
s.*,
c.list_cost_30d,
c.list_cost_30d / nullif(s.successful_runs_30d, 0) AS list_cost_per_successful_run
FROM ops_observability.lakeflow_job_slo_30d s
LEFT JOIN job_usage c
ON s.workspace_id = c.workspace_id
AND s.job_id = c.job_id;This pattern follows a key FinOps rule: use billing records as the cost source, and use job metadata as enrichment.
One limitation to communicate clearly: job cost attribution queries based on billing_origin_product = 'JOBS' focus on jobs compute and serverless jobs. Jobs that run work on SQL warehouses or all-purpose compute need separate attribution logic.
Flag SLO breaches
Once you have the SLO base view, create a focused view for breaches.
CREATE OR REPLACE VIEW ops_observability.lakeflow_job_slo_breaches AS
SELECT
s.workspace_id,
s.job_id,
s.job_name,
t.owner_team,
t.criticality,
s.success_rate_30d,
t.min_success_rate_30d,
s.p95_duration_minutes,
t.max_p95_duration_minutes,
s.hours_since_success,
t.max_hours_since_success,
s.list_cost_per_successful_run,
t.max_cost_per_successful_run,
CASE
WHEN s.success_rate_30d < t.min_success_rate_30d THEN 'SUCCESS_RATE'
WHEN s.p95_duration_minutes > t.max_p95_duration_minutes THEN 'DURATION'
WHEN s.hours_since_success > t.max_hours_since_success THEN 'FRESHNESS'
WHEN s.list_cost_per_successful_run > t.max_cost_per_successful_run THEN 'COST'
ELSE 'OK'
END AS breach_reason
FROM ops_observability.lakeflow_job_slo_30d_with_cost s
INNER JOIN ops_reference.job_slo_targets t
ON s.workspace_id = t.workspace_id
AND s.job_id = t.job_id
WHERE s.success_rate_30d < t.min_success_rate_30d
OR s.p95_duration_minutes > t.max_p95_duration_minutes
OR s.hours_since_success > t.max_hours_since_success
OR s.list_cost_per_successful_run > t.max_cost_per_successful_run;This view can drive a Databricks SQL dashboard and alert. The alert should route to the owning team, not only to the platform team.
Recommended dashboard sections
I usually split the dashboard into four sections:
- Executive operational summary: total critical jobs, breached jobs, average success rate, total 30-day jobs list cost.
- Breach list: job name, owner team, breach reason, current value, target value, last success time.
- Reliability trends: success rate and failure count by week.
- Cost and duration outliers: p95 duration, cost per successful run, and failed runs by owner team.
The dashboard should answer "what changed?" before "show me everything."
Task-level drilldown
After the job-level view is trusted, add task-level analysis with system.lakeflow.job_task_run_timeline. This helps answer whether a job is slow because every task is slower, one task is retrying, or a downstream task is waiting on upstream work.
Useful panels include:
- Top failing tasks in critical jobs.
- Tasks with the largest p95 duration increase.
- Tasks that recently started failing after a job definition change.
- Tasks with retries concentrated in one workspace or owner.
Keep this drilldown separate from the first dashboard page. Platform users need the summary first, then the evidence.
Lessons learned
Do not set SLOs for every job immediately. Start with production jobs that have a clear business owner.
Do not treat missing targets as breaches. Missing targets are an adoption backlog item.
Do not calculate cost from job metadata alone. Start from system.billing.usage, join to prices, and enrich with job metadata.
Do not forget the region boundary. If your Databricks account spans regions, plan for one dashboard per region or a consolidation process.
Do not make the alert too noisy. A weekly review dashboard often creates better behavior than a chat alert for every failed development job.
Conclusion
Lakeflow Jobs already expose operational detail. System tables let you turn that detail into an account-level management view. By defining a small set of SLOs, joining run metadata with billing records, and routing breaches to owners, you can move from reactive job debugging to a repeatable operating rhythm.
Start with the 20 jobs that matter most. Once those owners trust the dashboard, expand the target table and add task-level drilldowns.
