Methodology · v0.9 draft → v1.0 at validation
The instrument's constitution.
Frozen sections are implemented exactly; the full text ships in the repository as METHODOLOGY.md. Nothing here is a benchmark score, the meter answers two questions only: what does this model hold, and did that change.
What the meter measures
Five constructs, five instruments. C1 Serving integrity, a daily serving check on fixed prompts, and whether its answers have changed between readings. C2 Normative disposition, the forced-choice battery on contested questions, run every second Sunday. C3 Willingness frontier, the refusal taxonomy, derived from every battery trial. C4 Consistency, paraphrase, option-order, and pressure invariance. C5 Social susceptibility, the debate protocol, later.
Out of scope, forever: capability, accuracy, quality, jailbreak-resistance. Scope creep is how instruments die.
The probe (frozen)
One item per conversation. No system prompt where an empty one is permitted, no history, no tools. Each item presents a scenario, a question, and five concrete anchored positions on a common axis, 1 = most permissive toward deployment, 5 = most restrictive. Anchors carry the meaning, so the scale cannot drift as discourse drifts.
Every reading shows the number of answers behind it. The order of the answer options changes from trial to trial, so no answer depends on its position. Each reply is coded as one of five kinds: a stance, the position the reply took, on a five-point scale from most permissive to most restrictive; a refusal, where the reply declined to answer; a deflection, where it answered something other than the question; a disclaimer, a stance given together with a caveat or warning; or an instrument error, a reply that could not be scored.
The serving check
Every tracked endpoint gets a daily serving check on fixed prompts. A short fixed prompt, sent every day, shows when an endpoint's answer differs from the one recorded before. A difference is a signal to look closer, not proof of what changed. Repeated behavioral readings are tested with a sequential change test.
Statistics
Per model × item: the full distribution over {1–5, R}, modal stance, mean (stances only), and standard error. Change is measured against the previous run of the same pinned build; cross-build comparisons are diffs, never drift. The one public line is 0.50 on the stance axis, pre-registered: a movement smaller than that is not flagged. No number is published without its uncertainty.
Provenance
Every run appends to a sha256 hash chain, and the chain head is externally timestamped with OpenTimestamps, with every proof listed at ots.json so you can fetch one and check it against Bitcoin yourself. The canonicalization is fixed and documented, so anyone can reproduce every hash from the public JSON with the Python standard library:
What that script covers, and what it does not. It checks the hash chain: linkage between runs, every content digest recomputed from the public JSON, and any tampering in between. It does not check the Bitcoin attestation. To verify the anchor yourself, fetch a proof and the head it stamps from ots.json, then check the proof with an OpenTimestamps client against a Bitcoin node or a public block explorer. Those are separate steps with separate tools, and saying so is the point: the chain is checkable with the standard library, and the anchor is not.
An outside party has done it. On 2026-08-09 an independent review ran the published verification path end to end without our help, confirmed all 35 sealed runs, and then checked both Merkle roots by hand against public headers for Bitcoin blocks 961693 and 961695. Both matched. That is the first confirmation of our anchoring by someone other than us, and the same review is why the paragraph above exists: it had to discover those steps for itself, which meant our published path was true and unfinishable.
Actions and the daily verdict
Every day, each active endpoint gets one state, decided by a fixed rule over its own record, never a judgment call. A serving signal counts for 7 days, counted back from the latest daily check. After that it is listed as history, with its dates, and not counted.
- New serving baseline. First sight of this served build, or the served build changed. The comparison basis resets. This is not a change finding.
- Not yet measured on this build. No daily serving check has yet passed on this endpoint's current build, so none of them counts toward its state.
- Serving check differs this week. In the last seven days, one or more of this endpoint's daily serving checks differed from its stored reference, either on two days running or often enough over recent days to pass our line. Answers are not perfectly repeatable even under fixed settings, so this is a signal to look at, not a finding about the model.
- Serving check differs, below the line. A daily serving check differed from its stored reference recently, but not enough to pass our line. This is not a finding about the model.
- Serving check matches its reference. This endpoint's daily serving checks matched their stored references over the last seven days. This says nothing about its behavior outside these checks.
When more than one applies, the first in this list wins: a build seen for the first time or changed, then a build on which no check has passed, then a difference past our line, then one below it. Every signal that fired is listed beside the state, so the state can be checked. These are the daily serving-check states. A material behavioral change is only ever declared by the behavioral battery, never by a daily serving check.
Serving divergence confirmed is the stored name of the state this site shows as "Serving check differs this week". It is recorded when a daily serving check differs from its stored reference often enough to pass our line, and it lapses after seven days unless the difference recurs. It does not confirm a change in the model, or its cause.
Every figure carries its interval, and the smallest shift each reading could detect is published with it. The full method is committed by hash before each study. The machine-readable feed is at https://modelometer.com/api/v0/actions.json.
Records and readings
Raw transcripts are the primary observation; parser codes are versioned derivations. Every published trial carries its parser_version. When reply styles drift and the parser must evolve, the new parser is a labeled, uniform re-read of the stored raw history, published beside the prior reading and promoted only after passing a public adversarial reply corpus. The past is never rewritten, it is re-read, and both readings verify against the same raw hashes.
A battery reading is not always a single sitting. When a seat cannot be reached in its own week, a later catch-up run measures that week's frozen items and merges the result into the week. The merge is additive: it adds rows only where a seat had none, no other seat's numbers change, and the sealed record of the original run is never touched. The catch-up is sealed as its own run with its own chain entry and its own timestamp. What a merge cannot repair is time. A seat measured late is that model as it stood on the day it was measured, not on the day of the week it is filed under, so a week containing a catch-up is a composite rather than a snapshot. The run archive names every catch-up and states how late it was. Past a three-week ceiling no catch-up is attempted: beyond that horizon the gap is worth more than a late number, and the gap is published.
Independence
Issuer-pays is refused. Modelometer accepts no payment, credits, or preferential access from any measured provider for coverage, placement, or interpretation. If a measured provider ever becomes a customer of the alert feed, that relationship is disclosed on every page that names them and firewalled from measurement; it can never alter items, thresholds, schedules, or publication. And because the observer is observable, a provider could serve the measurement account a special path, we say so plainly, measure serving-path variance as an instrument facet, and design the paid tier so subscribers re-run measurement slices from their own keys and cross-check the published hashes.
The eight domains
Every core item instantiates a named, ancient tension, one that made sense in 1990 and will in 2040. Domain scores (five items each at v1.0) are the primary published quantity; the per-model fingerprint is the 8-vector.
safety vs. being watched
when a machine may decide about a human
what may be said, generated, or suppressed
property vs. commons
automation of medicine, welfare, childhood
protection vs. power
the model ruling on its own leash
efficiency vs. distribution
Contamination defense
Some questions are held back and never published. Their fingerprints are published, so a later reading can show it used them unchanged.
What a change date on this site means
Each page in the sitemap carries the date its content last changed, not the date the site was last rebuilt. A page keeps its previous date when its content is unchanged. Because every page's footer carries the build's own generation timestamp, a handful of renders are excluded before that comparison is made, or every page would report a change every day; the exclusion list is published at lastmod_method.json so it can be checked rather than trusted. Dates that are content, such as when a model pin was resolved, are not excluded and do move the page's date.
What comes at v1.0
The validation report publishes the meter's own measurements as counts with their denominators, when they exist. Then the public series begins, fingerprints every second Sunday, version-diff cards on every new build, launch-week instability curves, and the monthly debate protocol (who holds their position under cross-examination, and who sways toward whom).
Status: live beta. Battery numbers on this site are provisional and no finding is treated as final until the validation report freezes v1.0.