Why Your Solar Forecasting Model Doesn't Trust Every Zero

A data quality pattern for sensor dropout at IoT scale, using Lakeflow Declarative Pipelines (formerly DLT)

A solar panel producing zero output at 2pm on a clear day is a maintenance ticket. A solar panel producing zero output because it's midnight is completely normal. And a solar panel producing zero output because its inverter dropped off the network an hour ago and is now silently repeating a stale cached value — that one will quietly corrupt your forecasting model unless your pipeline is built to catch it.

This is the problem we ran into building a solar generation forecasting platform for a distributed energy retailer in New Zealand managing 2,400+ solar sites. Getting this distinction wrong had two failure modes, both expensive: false maintenance alerts burying the operations team in noise, or genuine faults going invisible until a customer called in. The fix wasn't a smarter model — it was teaching the pipeline itself to tell these cases apart before any forecasting model ever saw the data.

The problem, precisely

A "zero" reading from a panel can mean one of three completely different things:

• Genuine zero — nighttime, or heavy overcast. Expected, valid, and important signal.
• Fault zero — the panel is online and reporting correctly, but something is actually broken (inverter fault, shading, degradation). This is real signal too — it's the maintenance alert you want.
• Dropout zero — the sensor has gone offline and is reporting a stale or default value that only looks like a real reading.

Here's the part that makes this a genuinely hard problem: you cannot tell these apart from the value alone. A zero is a zero. What distinguishes them is context — what time it is, what irradiance should be expected at that location right now, whether neighboring sites are seeing similar conditions, and whether the sensor has actually been communicating recently. This is a labeling problem hiding inside what looks like a data engineering problem, and it's exactly the kind of thing that's invisible until it's already corrupted three weeks of training data.

Architecture, briefly

The platform runs on a fairly standard medallion architecture: Auto Loader ingests streaming telemetry from thousands of sites into Bronze, and Lakeflow Declarative Pipelines (the current name for what most of us still call DLT) enforces data quality gates on the way into Silver. Downstream, a champion/challenger setup — NeuralForecast's NBEATS against LightGBM — consumes clean, feature-store-backed data for 72-hour-horizon forecasting. The architecture itself isn't the interesting part. What happens at the quality gate is.

The naive approach (and why it fails)

The first instinct most teams have is a single expectation:

@dp.expect_or_drop("non_negative_output", "output_kw >= 0")

This catches genuinely malformed data, but it does nothing for the actual problem: it can't distinguish a dropout-zero from a legitimate nighttime zero, because both satisfy output_kw >= 0. Drop too aggressively and you lose real nighttime signal your model needs. Don't drop at all and corrupted dropout data flows straight into your feature store. Neither option works, and this is the point where most quick fixes stop — because the fix isn't a better threshold, it's a better question.

The quarantine pattern

Instead of asking "is this value valid on its own," the pipeline asks "is this value plausible given what else is true right now" — specifically, expected irradiance for that site and time, and how recently the sensor has actually communicated.

from pyspark import pipelines as dp
from pyspark.sql import functions as F

DROPOUT_CONDITION = (
    "output_kw = 0 "
    "AND expected_irradiance > irradiance_threshold "
    "AND minutes_since_heartbeat < 15"
)

# Main flow: only physically implausible readings are held back
@dp.table
@dp.expect_or_drop("plausible_output", f"NOT ({DROPOUT_CONDITION})")
def panel_readings_clean():
    return spark.readStream.table("panel_readings_bronze")

# Quarantine flow: capture exactly what the main flow rejected —
# nothing is silently discarded
@dp.table
def panel_readings_quarantine():
    df = spark.readStream.table("panel_readings_bronze")
    return df.filter(DROPOUT_CONDITION)

The logic: a zero is only routed to quarantine when it coincides with daylight-level expected irradiance and a recent heartbeat — meaning the sensor claims to be online and communicating, but is reporting something physically implausible for the conditions. Genuine nighttime zeros never touch this path at all; they pass straight through as valid signal. Known-offline sensors (no recent heartbeat) also don't get quarantined as anomalies — they're a separate, already-understood state.

One thing worth calling out: Lakeflow's built-in event log records every expectation violation automatically, so quarantine volume and rejection metrics are available for a data quality dashboard without hand-rolling any observability layer.

Closing the loop: what happens to quarantined data

Routing bad rows to a side table is the easy half of this pattern. The part that actually matters is what happens next — because simply discarding quarantined rows creates gaps that degrade a 72-hour forecast horizon just as much as bad data would.

The resolution here uses neighboring-site correlation: sites in similar geographic and electrical proximity see similar irradiance conditions, so a quarantined reading gets imputed from the correlated output of nearby, currently-healthy sites rather than left as a gap or replaced with a naive average. Every imputed value carries a confidence flag downstream, so the champion/challenger forecasting models can choose to weight, discount, or exclude imputed points rather than treating them as equal to a real reading. Enforcing this distinction at the pipeline level — not the modeling level — means every downstream consumer of the Silver table inherits the same guarantee, instead of every model author having to solve this problem independently.

Results

None of this is abstract — it's the reason a few concrete numbers were achievable in production:

• 18% reduction in grid over-purchase cost
• Sub-8% MAPE at the 72-hour forecast horizon
• 340+ proactive maintenance alerts surfaced in a single quarter
• Fault-to-fix time down from ~11 days to ~3 days

None of these numbers are reachable if the pipeline can't first tell a broken sensor from a quiet one. The forecasting model gets the credit, but the quarantine gate is what made the training data trustworthy in the first place.

Beyond solar

This exact ambiguity — a sensor reading of zero (or any "boundary" value) that could mean nothing happened or something is broken — shows up anywhere IoT-scale telemetry meets physical systems with expected idle states. Manufacturing line sensors during a planned changeover, HVAC systems overnight, connected-vehicle telemetry when a vehicle is parked — all of them have the same three-way ambiguity, and the same fix: don't ask if a value is valid in isolation, ask if it's plausible given context you already have elsewhere in the system.

Closing

The quarantine pattern itself is a few lines of Lakeflow code. The actual engineering work is deciding what context makes a boundary value suspicious in your domain, and building the imputation logic that keeps quarantined gaps from becoming forecasting gaps.

Have you run into a similar "the value looks fine but the context says otherwise" problem in an IoT or sensor pipeline? I'd love to hear how you solved it.