M&V tutorial · control group
Control groups: measuring a VPP against homes that sat out
The most accurate way to measure a virtual power plant (VPP) of homes is to compare called homes with similar homes that were not called, on the same day. Here is how randomized control groups and matched comparison groups work, and what each costs.
VPP Economics has no affiliation with, sponsorship from, or endorsement by any utility, aggregator, platform, or device maker named here.
The short answer
- Compare called homes with similar homes that were not called, in the same hour and the same weather. A randomized control group holds back a random share of enrolled homes; a matched comparison group finds look-alikes among homes that did not enroll (CAISO baseline working group).
- Randomized control groups are generally the most accurate option for home fleets. CAISO’s working group found them more than twice as precise as typical-day or weather baselines at 200 to 400 homes (CAISO).
- They see what history-based baselines miss. Extreme event days, pre-cooling and snapback all show up because the control group lives through the same day (ACEEE).
- The cost is held-back capacity, or hidden bias. Randomizing means some enrolled homes sit out each event. Matching avoids that, but no matching can rule out hidden differences in who chose to enroll (LBNL).
How a randomized control group works
Hold some homes back, compare the rest
A real example: PG&E SmartAC, June 25 event, 97.3°F.
Swipe to see the full chart →
A serial number has nothing to do with how a home uses electricity, so the held-back groups are, on average, the called homes without the event. No history, no weather model. Subtract one average from the other, multiply by homes called: 0.57 kW × 12,095 ≈ 7.0 MW. Rotating which groups sit out means every home is still dispatched on other events.
The event day, both groups
Average load per home for called and held-back homes.
Swipe to see the full chart →
Randomized or matched
Two ways to build the comparison
Both answer the same question: what would these homes have used without the event?
Swipe to see the full chart →
Randomization is the gold standard because it balances everything, measured or not. LBNL puts it plainly: with a matched group, however well the match appears to work, the possibility of unobservable differences producing bias cannot be ruled out (LBNL). Matching is still common because many programs never held anyone back. SDG&E relies on matched groups for exactly that reason (SDG&E).
What a control group sees
Before, during and after an event
Puget Sound Energy smart thermostat program, winter 2022–23, electric participants, per home.
Swipe to see the full chart →
A typical-day baseline would struggle here. Event days are colder or hotter than the days the baseline draws on, and a pre-heated house shifts load into the hour before. Because the comparison group lives through the same day, the 0.73 kW added before and the 0.84 kW added after are measured, not assumed. For smart thermostats, control groups are the most common evaluation method, with an average impact of 0.54 kW per participant across published reports (ACEEE).
The trade-off: held-back capacity
Precision against capacity held back
A 10,000-home fleet, 0.57 kW per home. More homes in the control group narrow the error but sit out more of each event.
Swipe to see the full chart →
Most of the precision comes early. Moving from 150 homes to 10% of the fleet cuts the error from about ±19% to ±8%; going further buys little. The capacity cost rises in a straight line. That is why large programs rotate small holdouts rather than keep a fixed group on the sidelines.
Who uses it
Settlement option for demand response
At least 150 distinct homes or businesses, not dispatched, with nearly identical demand patterns and the same weather. Residential and non-residential cannot be mixed (CAISO BPM).
Smart Cycle and Summer Cycler
Both run as randomized controlled trials, with treatment and control assigned before the season (CenterPoint).
AC Saver and ELRP evaluations
Matched comparison groups, because no group was withheld from dispatch (SDG&E).
Control groups are also the only practical way to measure behavioral demand response, where a text message, not a device, asks homes to cut back. A 75,000-customer randomized trial at PG&E measured peak reductions of about 2 to 3%, too small to see against any history-based baseline (ACEEE).
Where it goes wrong
Too few comparable homes
Small fleets cannot spare 150 homes, and matching needs a large pool of similar non-participants (ACEEE).
Matching on the wrong things
People who enroll may differ in ways no meter shows: who is home, comfort tolerance, other devices.
The control group is treated too
Control homes in another program, on a time-of-use rate or reacting to a public alert reduce too, shrinking the measured impact. CAISO drops other program event days in validation (CAISO).
The fleet changes
New enrollments and dropouts break the balance. CAISO requires revalidation when the mix changes by more than 10% (CAISO).
How it shows up in the VPP Program Designer
The VPP Program Designer offers both in Step 5. A randomized control group carries no bias, and its error shrinks with the square root of the control homes. A matched comparison group carries a small assumed bias with ±8% (residential) to ±12% (C&I) accuracy. Both are estimates; real precision depends on fleet size, holdout share and how alike the homes are.
All six ways performance is measured
Baseline from interval meter
Past meter data, sometimes with weather, predicts what the site would have used.
See the tutorial →For short eventsMeter before / meter after
Usage just before an event against usage during it.
See the tutorial →For fleetsStatistical sampling
A metered sample stands in for the whole fleet.
See the tutorial →For batteriesMetering device output
Measures what the device itself delivers.
See the tutorial →For firm loadsMaximum base load
Pays for staying below a pre-agreed level.
See the tutorial →For residential programsControl group
You are here. Compares dispatched homes with homes held back.
This pageMethod and caveats
A tutorial for program designers, utilities, aggregators and other readers. It is not a substitute for a tariff, manual or evaluation plan. The PG&E and Puget Sound Energy figures are as published; load shapes, held-back group choice and the trade-off curve are illustrative. Checked October 23, 2026.
The trade-off curve uses margin = 1.645 × σ × √(1/ncontrol + 1/ncalled), with σ = 0.8 kW after difference-in-differences, divided by the 0.57 kW impact.