# Result Benchmark Method 1.0

Decision date: 2026-10-28. Authority: the founder's request in this conversation to define
Result scoring as a real benchmark and update the product plan and examples.

**Status: acquisition calculation and campaign API implemented; real-data admission and production qualification pending.**
The website demonstrates the arithmetic using simulated reference data. No live reference set
has been admitted. Existing API Results remain measurement records; a separate campaign benchmark endpoint
returns the score or its withholding reason. The first implementation is acquisition only. This document does not change contract-1.0 or grant publication authority.
See [ADR 0006](../adr/0006-result-benchmark-scores.md).

## Meaning

The Review score judges the idea before action. The Result score benchmarks measured performance
after action. It uses neither the Review nor the customer's target. A Result score can be
computed without a Review. Neither score is a probability, and subtracting them is not an
improvement measure: their scales answer different questions.

The Result score is a **0–100 benchmark index**: a fixed weighted average of the percentile
positions of its required outcome measures within one named, frozen reference set. Higher is
better. A dimension at 50 sits at the reference middle. An overall 80 means an average dimension
position of 80 under the method's weights; it does **not** mean beating 80% of whole campaigns,
80% target attainment, an 80% success probability, or a universal rank across domains.

The raw measures, component scores, weights and comparison conditions remain visible. Equal
headline scores can conceal different strengths, so comparisons must expose that breakdown.
Meeting an owner's target is separate context and never enters the benchmark calculation.

## Freeze the benchmark before the outcome

NeoNumber publishes and versions a domain-specific method and a reference-set manifest. A
customer may select a relevant admitted method before starting, but cannot alter its measures,
weights, reference membership or eligibility rules and retain the standard benchmark label.
An eligibility resolver assigns the narrowest matching published stratum using input conditions,
not the eventual outcome. Ambiguous or unsupported matches have no headline score.

Each manifest must contain:

- Method ID/version, reference ID/version/content digest, owner, admission record and expiry.
- The exact population, inclusion/exclusion rules, measurement unit, geography/currency where
  relevant, product and audience class, scale/spend bands, starting conditions, measurement
  horizon, calendar window, evidence track and live/paper/backtest designation.
- Each required measure's counting rule, denominator, direction (higher/lower is better),
  precision, minimum exposure/sample, observation window, declared uncertainty estimator and
  collection deadline. Normalization uses recorded exposure, never the owner's goal.
- All required dimensions and fixed positive weights summing to one. Measures must not silently
  double-count the same outcome. A written rationale and sensitivity analysis justify weights.
- Non-compensable conditions, such as complete delivery, zero unresolved integrity failures or
  an explicitly required safety measure. A failure retains component results but withholds the
  overall score with its reason; strong performance elsewhere cannot offset it.
- The eligible, observed, excluded and missing counts, exclusions by reason, owner concentration,
  recruitment/participation limits, measurement limitations and reference membership receipt.
- Every reference observation or an access-controlled reproducibility receipt, its source class,
  exact implemented version and audit trail. Publication rights are separate from private use.

The registered work pins the method, reference and conditions before work starts. References
exclude the subject, its owner and controlled affiliates. Multiple arms/versions from one
experiment count as one unit under a rule fixed in advance; no selecting its winning arm.
If removing a subject's group breaks an admission floor, that subject has no benchmark score.

A historical reference may cover at most the previous 12 months and expires no later than 12 months after
its cutoff; a domain can require a shorter window. Later reference versions apply prospectively.
Old scores keep their original references. Recalculating against a newer reference creates a
separately labeled comparison, never a silent revision. Retrospective submissions keep their
measurements and "added after completion" history, but receive no standard headline in v1.

Forecast series use a prospective comparison panel instead of historical outcomes. Before the
first event locks, freeze the event set, participant roster, eligibility rules, resolution rules,
deadlines and weights. Each participant then commits a probability before each event's lock.
Reference losses become available only when the shared events resolve; seal their completed
manifest before calculating ranks. Never recruit or exclude participants based on their losses.
The same admission, completion and ownership rules apply, and missing series remain in coverage
and sensitivity reporting. This panel is valid only for its registered event set; the historical
12-month window does not substitute different past events. Its digest records both the initial
registration and the completed observations, without pretending those observations existed early.

## Admission and the cold start

Default minimums are **100 complete independent work units from at least 10 unaffiliated
owners**, no owner/control group above **10%**, and observed outcomes for at least **80%** of
the eligible registered units in the reference window. Apply these after exclusions. The unit
is a campaign, experiment or forecast series, not a click, ticket, forecast event or repeated
measurement. These are conservative product-policy floors, not a statistical proof of quality.
A domain may tighten them; changing them requires a new method version and recorded decision.

An admission review must additionally establish comparability, legal reuse, measurement quality,
non-selective recruitment and follow-up, adequate exposure, uncertainty estimates and robustness
to missing outcomes and weights. More records cannot repair a biased or mismatched population.
All known eligible work, including misses, must enter the registry before outcomes; success-only
submissions cannot qualify. No missing outcome is converted to zero or quietly discarded.

References are scoped samples, not the whole market. The displayed claim is "compared with
[named reference]", never "better than other businesses". Keep independent/connector-attested
and customer-reported evidence in distinct reference tracks. A reported-data score must say
"based on customer-reported data"; it does not become verified by being benchmarked. Backtests,
simulations, affiliated work and live unaffiliated observations cannot share a real reference.

Until a reference is admitted and the subject meets every condition, show **"Not enough
comparable results yet"**, a concrete reason and the available measurements. Withhold the point
score and band. Missing required subject measures also withhold the overall score; show eligible
component scores without reweighting away missing or unfavorable dimensions. No average,
industry-looking anchor or invented percentile substitutes for absent benchmark evidence.

## Calculation

For required measure j with observation x, let N be the complete frozen reference count:

    P_j = 100 × (number strictly worse than x + 0.5 × number tied with x) / N
    Result score = sum(weight_j × P_j)

For higher-is-better measures, reference values below x are worse; for lower-is-better measures,
values above x are worse. Ties use the method's frozen measurement precision, not a new tolerance
selected after seeing the data. An all-tied measure returns 50. An observation below/above the
entire reference can return 0/100; 100 means above this reference, not perfect or risk-free.

Use exact decimal arithmetic on declared-precision inputs, retain unrounded component/index
values, and round the displayed headline HALF_UP to the nearest integer only at presentation.
Never average displayed rounded scores. Equal weights are a transparent initial choice, not a
claim that dimensions contribute equally to business value.

Example: 80 reference campaigns have worse acquisition cost than the subject and none tie:
cost position = 80. Its refund-rate position is 60. With fixed 50/50 weights, Result = 70.
Changing its personal acquisition target from $25 to $50 changes neither position nor Result.
Different weights would define a different method, not a preference toggle on this Result.

## Uncertainty and robustness

The rank inside a frozen finite reference is reproducible; extrapolation beyond it is uncertain.
Every real headline must carry a reference-sensitivity band and measurement limitations.
Use 10,000 owner-cluster bootstrap replicates (resample owners with replacement, include each
drawn owner's complete units, recompute component ranks and the weighted index). Record the
generator/version/seed and the exact resampling-index digest. Admission floors apply to the
original reference, not each bootstrap replicate. Use empirical 2.5% and 97.5% order statistics
(nearest-rank convention) and round outward. This is a descriptive reference-sensitivity band,
not a claim of representative market coverage or causal confidence.

For measured quantities with declared intervals, evaluate the monotone index at all-worst and
all-best dimension endpoints in each replicate; take the outer envelope with the reference-only
band. If a required measure cannot support its declared uncertainty estimate, withhold the
headline. Never replace an unknown interval with a zero-width band. The synthetic website
examples deliberately show no statistical band: simulated ranks cannot establish uncertainty
about real populations.

Before admission, also publish leave-one-owner-out results, each weight changed by ±20% and
renormalized, and bounds assigning missing reference outcomes all-better/all-worse positions.
Do not use these scenarios to tune for a favorable answer. A sensitivity range exceeding 20
points, or a reversal of a displayed comparison under these checks, blocks an unqualified
headline/comparison until the domain method resolves the cause. The threshold is product policy.
Overlapping bands do not support a "better" claim; report the difference as inconclusive.

## Domain profiles

These definitions specify the measures for the illustrations and the starting profiles for
qualification. They are not admitted live benchmarks; full eligibility manifests are required.

| Profile | Required measures and weights | Conditions that must match |
|---|---|---|
| Acquisition campaign | 50% cost per eligible new customer (lower); 50% refunded-order fraction (lower) | Same acquisition channel, offer/price band, new-customer definition, market, spend/exposure band, 30-day acquisition window and completed 14-day refund follow-up; total allocated acquisition cost and all eligible first orders before refunds |
| Product onboarding | 50% first-design completion fraction (higher); 50% seven-day return fraction (higher) | Comparable design products, same new-visitor denominator, traffic source/device mix, two-week intake, seven-day follow-up; predeclared minimum unique visitors |
| Subscription pricing | 50% net monthly recurring revenue change (higher); 50% existing-subscriber cancellation fraction (lower) | Same subscription/customer stage, price band, baseline revenue period, 60-day horizon and accounting for discounts/refunds; frozen eligible subscriber population |
| Four-day-week pilot | 50% on-time delivery fraction (higher); 50% median customer first-reply hours (lower) | Same team function/size, workload/difficulty band, service hours, eight-week horizon and baseline; work scope frozen before pilot, abandoned work remains in denominator; complete support queue |
| Forecast series | 100% mean proper loss (lower) on an identical frozen event set | Same binary Questions, resolution versions, lock times and weights; at least 30 resolved events with complete pre-outcome forecasts for every eligible forecaster; reference unit is a forecaster's full series |
| Annual-plan launch | 50% eligible paid annual orders before refunds / original waitlist (higher); 50% refunded orders / eligible orders before refunds (lower) | Same annual subscription/price/offer band, warm waitlist definition, channel, waitlist-size band and 30-day acquisition window; mature 14-day refund follow-up |

Acquisition and launch conversion count eligible orders before refunds; refund rate is the
separate quality measure. This avoids penalizing a refund in both components. Net retained
sign-ups can still be a personal goal. Rates in the launch illustration round HALF_UP to
0.01 percentage point before ranking: 245 / 4,000 becomes 6.13%, and 9 / 245 becomes 3.67%.

The personal targets in an intake remain useful but have zero weight in these calculations.
Observed relative performance does not prove the intervention caused it; incremental claims
still require the identification rules in [Measurement Methods](measurement.md).

## Forecasts retain proper scoring

For a binary event, keep Brier loss `(p-y)^2` as the underlying event score; for a series take
the frozen equal-weight mean over the same events. A probability of .70 followed by y=1 gives
loss .09. One such result is an event score, not a calibrated track record or series benchmark.
Do not pool binary, categorical, interval or different-horizon loss functions into one ranking.

Apply the percentile calculation to series mean losses only after reference qualification.
The resulting 0–100 position is a descriptive benchmark index, **not a proper scoring rule**.
Training, incentives, forecast elicitation and claims of statistical skill use the underlying
proper losses and paired event analysis, never the rank index. A user cannot choose an easier
event subset: missing forecasts/unfinished events leave the series unscored. The resolver remains
blind to commitments and the reviewer to Resolutions, as required by ADR 0002.

## Corrections, outputs and implementation boundary

Corrected subject data supersedes its Result under the same frozen reference. Corrected reference
data creates an explicitly superseding reference version, with affected Results identified and
reissued as corrections; original values and receipts stay readable. A correction never changes
weights, eligibility or objectives. Method improvements create a new method, not a correction.

An eventual additive API artifact must carry method/reference IDs, digests, registration and
implemented-version links, evidence track, eligibility state/reasons, raw measures, component
positions and weights, unrounded/displayed index, band and its meaning, coverage, correction
links, and the distinction between constructed and observed. Unknown/pending/insufficient are
states, never numeric zero. Do not squeeze this artifact into a contract-1.0 measurement member.

Implementation gates: additive contract design and correctness ceremony; production decimal
calculation and refusal tests; reference registry/admission and access rules; uncertainty and
sensitivity evidence; API/UI/export support; an admitted real reference; beta qualification.
This change delivers the specification and reproducible website illustrations only. It does
not assert those production gates passed or change existing runtime measurement behavior.

## Methodological sources and original choices

[OECD/JRC composite-indicator guidance](https://www.oecd.org/en/publications/handbook-on-constructing-composite-indicators_533411815016.html)
informs the need to declare normalization, weighting and sensitivity; see also the
[JRC sensitivity guide](https://knowledge4policy.ec.europa.eu/composite-indicators/toolkit_en/navigation-page/10-step-guide_en/step-8-sensitivity-analysis_en).
[Gneiting and Raftery (2007)](https://sites.stat.washington.edu/people/raftery/Research/PDF/Gneiting2007jasa.pdf)
supports retaining proper loss for honest probabilistic evaluation. These sources do not
validate NeoNumber's profiles, data floors, weights, bands or score. Those are original,
explicit product choices requiring empirical qualification. No external code was copied.

## Acquisition implementation profile (2026-10-29)

The [reference runbook](../campaign-reference-runbook.md) defines the narrow first profile and
operator admission process. Each reference declares at least 30 eligible new customers and 30
eligible first orders per unit, with documented measurement precision. These are added exposure
floors; they do not weaken the independent-unit, ownership or completeness floors above.
The implementation withholds if the combined measurement/reference band exceeds 20 points,
in addition to the preexisting sensitivity rules. This is a conservative precision gate, not
a claim that narrower bands establish representativeness. Complete delivery and an unambiguous,
non-provisional registered report are required. Other profiles remain specifications.
