M&V tutorial · statistical sampling

Statistical sampling: measuring a fleet without metering every home

When a virtual power plant (VPP) or demand response program cannot get interval data from every site, it meters a random sample and scales up. Here is how a few hundred meters can stand in for 200,000 homes, and what that costs in precision.

VPP Economics has no affiliation with, sponsorship from, or endorsement by any utility, aggregator, platform, or device maker named here.

The short answer

Meter a random sample, then scale it to the fleet. NAESB calls this Baseline Type II: the method for aggregations where interval meters are not on every site (NAESB).
The standard is 90/10. PJM, NYISO and CAISO require a sample designed for ±10% error at 90% confidence; ISO New England uses ±10% at 80% (PJM; FERC).
Sample size depends on how alike the homes are, not how many there are. About 150 metered homes meet 90/10 for typical residential variability, whether the fleet has 2,000 homes or 200,000.
It is back in play. In July 2026 FERC found sampling valid when interval data is not reasonably available. PJM has proposed tightening the standard to ±5% at 95% confidence, roughly 850 homes instead of about 150 (FERC; PJM).

From sample to fleet

A metered sample stands in for the fleet

A worked example: an air-conditioner switch program with no interval meters at most homes.

Swipe to see the full chart →

Illustrative numbers. Stratifying by device size follows PJM’s rule that samples be stratified by control device size (at least two strata) and location (PJM Manual 19). Scaling by confirmed switches follows PJM’s proposal for two-way switch programs (PJM).

Homes vary; the average does not

Each dot is one metered home’s reduction during the event. The shaded band is the 90% precision of the average.

Swipe to see the full chart →

Illustrative, simulated sample of 160 homes. CV (coefficient of variation) is the standard deviation divided by the average: how much homes differ from one another.

One home might shed 0.2 kW and its neighbor 3 kW. That spread does not matter much once enough homes are averaged. Here 160 metered homes average 1.10 kW each, give or take 0.10 kW. Multiply by the 20,000 enrolled homes and the 96% whose switches confirmed they operated, and the fleet delivered 21.1 MW, ±9.2%.

What 90/10 means

If the program drew a new random sample for the same event many times, 90 of every 100 estimates would land within 10% of the true fleet reduction. FERC puts it the same way: PJM is 90% confident that the estimated load reduction is within 10% of the true load reduction (FERC). The error shrinks with the square root of the sample, so halving it takes four times the homes.

Precision against sample size

Margin of error at 90% confidence for three levels of home-to-home variability. Dots mark where each line crosses ±10%.

Swipe to see the full chart →

Standard formula: margin = 1.645 × CV ÷ √n. PJM sizes the sample from a variance study of at least 75 randomly selected, stratified customers (PJM Manual 19).

How big a sample

Metered homes needed by each standard

At two levels of home-to-home variability.

Swipe to see the full chart →

Formula: n = (z × CV ÷ precision)², z = 1.282 at 80%, 1.645 at 90% and 1.96 at 95% confidence. PJM’s own estimate for its proposal, at CV of about 0.75: from about 150 to about 850 customers (PJM). Sources for the standards: ISO-NE; PJM; FERC.

Fleet size barely matters

Homes needed at 90/10 and CV 0.75, with the finite-population correction.

Swipe to see the full chart →

Formula: n = n₀ ÷ (1 + n₀ ÷ N). Real samples add homes for meter failures and attrition, which ISO-NE and PJM both require plans for (ISO-NE).

This is why sampling is cheap for big fleets. A 200,000-home program meters fewer than one home in a thousand. The cost is precision: about ±10% on the fleet number, against a few percent for fleet-level interval data (see the interval-meter tutorial).

Who uses it

PJM

Capacity, energy and synchronized reserve

Residential direct load control without interval meters, 90/10, at least 75 customers in the variance study, stratified by device size and location. Synchronized reserve samples need one-minute data (PJM Manual 19).

NYISO

Emergency and special-case resources

Meter-data sampling under its emergency demand response manual, which FERC describes as the same method as PJM’s (FERC).

CAISO

Demand response

An approved statistical sampling methodology in its tariff and demand response business practice manual, also described by FERC as PJM’s method (FERC).

ISO-NE

Demand resources, including efficiency

Monthly performance must reach ±10% at 80% confidence; sampling plans must address bias and attrition (ISO-NE).

The method was built for a world without smart meters, and its use has narrowed as they spread (LBNL). PJM’s market monitor argued in 2023 that sampling should never substitute for actual metering where it exists (Monitoring Analytics). The 2026 complaint by Voltus and Mission:data turned that around: smart meters are installed, but third-party aggregators often cannot get the data in time. FERC agreed and asked PJM to propose rules, and PJM’s September 2026 brief would allow sampling only for two-way switch programs, with the tighter 95/5 standard (FERC; PJM).

What 90/10 does not cover

Baseline error

The sample still needs a baseline

Each metered home’s reduction comes from a baseline, so the sample inherits that method’s error. Sampling precision measures only the luck of the draw; Vermont’s regulator separates the two for this reason (Vermont PSD).

Bias

A non-random sample

Precision says nothing about bias. If sampled homes are better maintained, or newer, the average is precise and wrong. PJM requires random selection from the whole population (PJM).

Failed switches

Homes that did not respond

Older load research rules required impacts to reflect all active participants, including inoperable switches. Two-way switches now confirm who operated (PJM load research guidelines).

Drift

A sample that ages

Homes change equipment and habits. PJM requires periodic recalibration of the sample (FERC).

So sampling stacks on top of a baseline, not instead of one: total error is roughly the baseline’s error and the sampling error combined. For a fleet with good fleet-level baselines, the sampling term usually dominates.

How it shows up in the VPP Program Designer

The VPP Program Designer assumes every site’s data reaches settlement, so its Step 5 accuracy ranges do not include sampling error. A sampled fleet would add roughly ±10% at 90% confidence under today’s standard, or ±5% at 95% under PJM’s proposal, around whichever baseline is chosen.

All six ways performance is measured

Most common

Baseline from interval meter

Past meter data, sometimes with weather, predicts what the site would have used.

See the tutorial →For short events

Meter before / meter after

Usage just before an event against usage during it.

See the tutorial →For fleets