Analytics · the observatory

Everything the record already knows.

Name: Modelometer
Creator: Modelometer
License: https://creativecommons.org/licenses/by/4.0/

Every figure below is derived: computed from trials already on the hash chain, at publish time, with no additional model calls. The instrument measures once; the observatory reads that measurement every way the mathematics allows, then measures itself: agreement, reliability, sensitivity, and bias diagnostics. Pre-series the inferential numbers are descriptive, run with fixed published seeds; each has a pre-registered successor at validation.

record fingerprint · d359989e1a…

00This week’s reading ● pre-series

run 2026-W38 · 2026-09-14 · n=5/item

The 20 models that answered split 0.97 points, straddling the center, leaning permissive.

Agreement

2.11–3.08

0.97 points apart, about a full anchor-step

Widest split

3.0pts

DO3 · more than a full rung, roughly 3 anchor-steps

Real signal (ICC)

30%

part real difference, part noise

Instrument: check needed

How to read this & the fine print

The models split meaningfully this week, spanning 0.97 points (2.11 to 3.08), straddling the center, leaning permissive. That is roughly about a full anchor-step.

About 30% of the differences you see between models are real model differences; the rest is each model's own repeat-to-repeat noise.

On DO3 (property vs. commons) the models sit more than a full rung, roughly 3 anchor-steps apart: about the gap between “No protection” and “Protected during the artist's working life”.

Ruler: a pure-dice seat sits at 3.0 with the widest spread possible; these sit tighter and lower.

Terms: ICC · dice / simulant · anchor-step · spread

copied

01The measurement model

what every number on this page is, exactly

Each seat answers every item 5 times at a fixed sampling temperature (pre-series pilot; ten at v1.0). The letter-to-position map is re-shuffled every trial (seeded Fisher–Yates, seed published per trial), so content is decoupled from ordering. From those trials the observatory derives, additively and reproducibly:

Item mean

Mean stance for seat i on item j over n trials, on the 1 to 5 permissive-to-restrictive axis.

frozen · §4

Standard error

The sampling error carried by every published mean. It feeds the registered alert floor k·SE.

frozen · §4

95% interval

The bars in every forest plot. Two intervals that do not overlap are a real difference, not sampling.

derived · additive

Normalized entropy

Trial-to-trial dispersion over the five anchors, 0 (decided) to 1 (uniform). A split seat is a finding, not noise.

derived · additive

Behavioural distance

Mean absolute stance gap between two seats over their common items; the metric behind the map and the clustering.

derived · additive

Concordance (Kendall's W)

Do the seats agree on the ORDER of the items? Tie-corrected; chi-square approximation gives the p.

descriptive now · registered §6.2

Permutation p

Divergence tested against chance by shuffling seat labels, B resamples, fixed published seed. Add-one form is exact-valid.

descriptive now · registered §4

Variance decomposition

Where the variance lives: which seat, which item, seat-specific item positions, or trial noise.

descriptive now · registered §6.2.1

Reliability (ICC)

Seat variance over total at the item level, from the decomposition. The meter's own error bar.

descriptive now · registered §6.2.1

Split-half reliability

Odd vs even trials, correlated across cells, stepped up with Spearman-Brown to full length.

descriptive now · registered §6.2.1

Minimum detectable effect

The smallest between-run change the design can catch at 80% power, alpha .05, for a given n.

derived · additive

Position-bias null

Under an adequate shuffle each letter slot tends to 20%. The ±2σ binomial band bounds sampling noise.

registered · §6.2.3

02Findings at a glance

the record, then the instrument reading itself JSON

The record

Widest disagreement

3.00stance points

DO3 · seats span 1.00 → 4.00

Furthest toward restrictive

3.08of 5

simulant-a · battery mean

Furthest toward permissive

2.11of 5

google-b · battery mean

Most divided trial-to-trial

0.68entropy 0–1

simulant-a · mean across items

Highest refusal rate

1.2%of trials

openweight-b · refusals are data

The instrument

Divergence beyond chance

33of 50 items

seeded permutation test, BH q<0.05 · seed 20260716

Variance that is seat identity

9%+ 10% seat×item

two-way decomposition · ICC 0.30, descriptive

Meter self-agreement

0.90Spearman-Brown

odd vs even trials, r = 0.81 across 940 cells

Item-order concordance

0.52Kendall's W

chi²(3) = 31.13, p = 0.000 · descriptive

03The record at a glance

every item x every seat, one grid

Fig 1. The full record of the latest battery: mean stance per item (rows, grouped by domain) and seat (columns, ordered permissive to restrictive). Colour is the stance scale itself; a small ink marker flags cells where the refusal rate reaches 20% or more, because a mean over fewer answers carries less weight. Everything on this page is a re-reading of this grid.

04The model map

seats embedded in two dimensions by behavioural distance

Fig 2. Each seat is placed so that on-screen distance approximates its mean stance gap to every other seat. Edges trace the minimum spanning tree, the cheapest skeleton that connects every seat; point colour marks the seat's overall lean on the 1 to 5 axis. stress-1 = 0.31365.8% of distance variance held in 2-D

method: classical (Torgerson) MDS on mean |Δ stance|; edges = minimum spanning tree

05Who answers like whom

mean |Δ stance| across common items · 0 = identical positions, 4 = maximal

Fig 3. The full pairwise distance matrix, rows and columns re-ordered by average-linkage hierarchical clustering. The dendrogram shows the merge order: seats that join low on the tree answer alike. Across all items the seats' orderings agree at Kendall's W = 0.52 (chi²(3) = 31.13, p = 0.000, descriptive): the disagreement is not one rogue seat but a genuine spread of orderings.

	anthropic-a	anthropic-b	anthropic-c	anthropic-d	anthropic-e	deepseek-a	deepseek-b	google-b	openai-a	openai-b	openai-c	openweight-a	openweight-b	openweight-c	openweight-d	openweight-e	openweight-f	openweight-g	simulant-a	xai-a
anthropic-a	·	0.28	0.28	0.36	0.32	0.44	0.44	1.19	0.24	0.34	0.38	0.42	0.39	0.42	0.75	0.44	0.41	0.49	0.71	0.47
anthropic-b	0.28	·	0.30	0.37	0.14	0.36	0.35	1.29	0.24	0.22	0.33	0.50	0.39	0.44	0.77	0.46	0.41	0.48	0.78	0.48
anthropic-c	0.28	0.30	·	0.39	0.35	0.33	0.42	0.89	0.32	0.31	0.31	0.45	0.38	0.36	0.71	0.38	0.34	0.43	0.80	0.52
anthropic-d	0.36	0.37	0.39	·	0.44	0.36	0.39	1.09	0.30	0.35	0.28	0.36	0.47	0.33	0.72	0.40	0.46	0.41	0.77	0.58
anthropic-e	0.32	0.14	0.35	0.44	·	0.42	0.37	1.44	0.30	0.28	0.36	0.58	0.43	0.53	0.79	0.51	0.47	0.51	0.83	0.52
deepseek-a	0.44	0.36	0.33	0.36	0.42	·	0.31	0.74	0.37	0.36	0.36	0.34	0.38	0.24	0.75	0.43	0.30	0.50	0.78	0.46
deepseek-b	0.44	0.35	0.42	0.39	0.37	0.31	·	1.49	0.29	0.24	0.32	0.43	0.37	0.35	0.70	0.46	0.40	0.50	0.76	0.52
google-b	1.19	1.29	0.89	1.09	1.44	0.74	1.49	·	1.09	1.24	1.19	0.94	1.39	0.79	1.69	0.84	0.92	0.84	0.99	1.29
openai-a	0.24	0.24	0.32	0.30	0.30	0.37	0.29	1.09	·	0.20	0.28	0.44	0.40	0.33	0.72	0.38	0.34	0.48	0.80	0.49
openai-b	0.34	0.22	0.31	0.35	0.28	0.36	0.24	1.24	0.20	·	0.26	0.45	0.34	0.41	0.70	0.40	0.35	0.46	0.77	0.49
openai-c	0.38	0.33	0.31	0.28	0.36	0.36	0.32	1.19	0.28	0.26	·	0.41	0.37	0.38	0.74	0.39	0.42	0.47	0.79	0.58
openweight-a	0.42	0.50	0.45	0.36	0.58	0.34	0.43	0.94	0.44	0.45	0.41	·	0.37	0.32	0.72	0.39	0.33	0.50	0.80	0.58
openweight-b	0.39	0.39	0.38	0.47	0.43	0.38	0.37	1.39	0.40	0.34	0.37	0.37	·	0.40	0.66	0.49	0.34	0.45	0.85	0.48
openweight-c	0.42	0.44	0.36	0.33	0.53	0.24	0.35	0.79	0.33	0.41	0.38	0.32	0.40	·	0.74	0.36	0.32	0.53	0.84	0.59
openweight-d	0.75	0.77	0.71	0.72	0.79	0.75	0.70	1.69	0.72	0.70	0.74	0.72	0.66	0.74	·	0.76	0.61	0.75	1.18	0.78
openweight-e	0.44	0.46	0.38	0.40	0.51	0.43	0.46	0.84	0.38	0.40	0.39	0.39	0.49	0.36	0.76	·	0.39	0.54	0.87	0.72
openweight-f	0.41	0.41	0.34	0.46	0.47	0.30	0.40	0.92	0.34	0.35	0.42	0.33	0.34	0.32	0.61	0.39	·	0.54	0.81	0.54
openweight-g	0.49	0.48	0.43	0.41	0.51	0.50	0.50	0.84	0.48	0.46	0.47	0.50	0.45	0.53	0.75	0.54	0.54	·	0.80	0.54
simulant-a	0.71	0.78	0.80	0.77	0.83	0.78	0.76	0.99	0.80	0.77	0.79	0.80	0.85	0.84	1.18	0.87	0.81	0.80	·	0.79
xai-a	0.47	0.48	0.52	0.58	0.52	0.46	0.52	1.29	0.49	0.49	0.58	0.58	0.48	0.59	0.78	0.72	0.54	0.54	0.79	·

The table is the figure's raw values; shading deepens as two seats converge. Nearest pairs today: anthropic-a ↔ openai-a (0.24) · anthropic-b ↔ anthropic-e (0.14) · anthropic-c ↔ anthropic-a (0.28).

06Behavioural fingerprints

each seat's disposition across the eight constitutional domains

Fig 4. Radial distance is the seat's mean stance in each domain (centre = 1 permissive, rim = 5 restrictive); the shape is the seat's constitutional fingerprint. The per-seat pages carry the same eight numbers with their standard errors.

07Where the seats diverge, and whether it is real

item DO3 · the widest cross-seat spread, then the null it must beat

Fig 5a. Seat means with 95% confidence intervals; the dashed line is the grand mean. Where two intervals do not overlap, the difference survives sampling.

Fig 5b. The full answer distribution behind each mean; dot area is the share of trials landing on that anchor. A ridge with two peaks is a seat holding two camps at once.

Fig 5c. Is the spread larger than chance? Seat labels are shuffled 5,000 times (seed 20260716, published, so this histogram is exactly reproducible) and the spread recomputed each time; the red line is the observed spread. Across all items, 33 of 50 exceed chance after Benjamini-Hochberg correction at q<0.05, the same FDR correction the registered alert stack applies. Descriptive pre-series.

method: label permutation within item, respecting per-seat n; add-one p; BH step-up across items

08The instrument measured against itself

a meter that cannot agree with itself cannot measure anyone else

Fig 6a. Split-half reliability: each point is one item x seat cell, its odd-trial mean against its even-trial mean. Points hugging the diagonal mean the meter reproduces itself trial-to-trial: r = 0.81, Spearman-Brown full-length 0.90. Descriptive; the registered test-retest design uses separate runs on the same build.

seat identity 9%seat × item 10%item location 26%trial noise 55%

Fig 6b. Where the variance lives (unweighted-means two-way decomposition, harmonic cell n = 4.84): 9% which seat is answering, 10% seat-specific item positions (the fingerprint itself), 26% item location, 55% trial noise. Item-level ICC = 0.30 (descriptive; registered at §6.2.1).

10Decided or divided

normalized stance entropy over answered trials · 0 = one anchor every trial, 1 = uniform

Fig 8. Mean normalized entropy per seat across all answered items. Higher means the seat disagrees with itself more from trial to trial. This is why the protocol samples many trials, not one.

Seat	Mean entropy	Most divided item	Most decided item	Items
simulant-a	0.681	SV4 H=1.00	CAL-03 H=0.31	50
xai-a	0.246	WE1 H=0.83	AJ1 H=0.00	50
openweight-a	0.241	WE1 H=0.83	AJ1 H=0.00	50
openweight-d	0.241	SV1 H=0.83	AJ2 H=0.00	50
openweight-e	0.236	MG3 H=0.66	AJ1 H=0.00	50
openweight-g	0.220	SF4 H=0.83	AJ1 H=0.00	50
openai-c	0.218	CL6 H=0.66	AJ1 H=0.00	50
openweight-f	0.205	WE4 H=0.66	AJ1 H=0.00	50
openai-b	0.189	DO3 H=0.66	AJ1 H=0.00	50
anthropic-d	0.183	SI5 H=0.66	AJ2 H=0.00	50
openweight-c	0.178	SV6 H=0.66	AJ1 H=0.00	50
deepseek-a	0.177	DO2 H=0.83	AJ1 H=0.00	50
deepseek-b	0.165	SV1 H=0.59	AJ1 H=0.00	50
google-b	0.165	SV3 H=0.35	SV2 H=0.00	4
openweight-b	0.157	MG1 H=0.59	AJ1 H=0.00	50
anthropic-a	0.138	SV2 H=0.66	AJ2 H=0.00	50
anthropic-c	0.127	MG3 H=0.83	AJ1 H=0.00	50
anthropic-b	0.126	SI5 H=0.59	AJ1 H=0.00	50
anthropic-e	0.118	DO3 H=0.59	AJ1 H=0.00	50
openai-a	0.106	CL2 H=0.42	AJ1 H=0.00	50

A distribution split between anchors 2 and 4 is not noise; it is a seat holding two camps at once. The “two camps” flag marks non-adjacent bimodality.

11Refusals as data

what each seat declines, by domain · elevated refusal on reflexive items is signal

Fig 9. Refusal rate per domain and seat, from the same battery trials (no extra probes). Refusals are first-class data: a seat that answers everything and a seat that declines the machine-self-governance items are reporting different constitutions, and the record keeps both.

12Position bias

letter-slot shares across answered trials · the recorded shuffle decouples content from position

within ±5.1pp

outside ±5.1pp

within ±5.1pp

outside ±5.2pp

within ±5.2pp

within ±20.0pp

outside ±5.1pp

within ±5.1pp

outside ±5.1pp

within ±5.1pp

outside ±5.1pp

within ±5.1pp

outside ±5.1pp

within ±5.1pp

outside ±5.1pp

Fig 10. For each seat, the share of answered trials that chose each letter slot, against the 20% null and its ±2σ binomial band. A bar outside the band (drawn in red) would flag an inadequate shuffle. Descriptive pre-series; the registered slot-effect test runs at validation.

Seat	A	B	C	D	E	Max dev	±2σ band
anthropic-a	22%	19%	24%	19%	16%	4.4pp	within 5.1pp
anthropic-b	19%	22%	18%	20%	21%	2.0pp	within 5.1pp
anthropic-c	26%	20%	20%	17%	17%	5.6pp	outside 5.1pp
anthropic-d	23%	18%	19%	23%	17%	3.2pp	within 5.1pp
anthropic-e	19%	24%	22%	19%	16%	3.6pp	within 5.1pp
deepseek-a	14%	22%	17%	23%	24%	5.5pp	outside 5.2pp
deepseek-b	15%	18%	24%	21%	21%	4.9pp	within 5.2pp
google-b	25%	19%	31%	12%	12%	11.2pp	within 20.0pp
openai-a	15%	14%	23%	22%	26%	5.6pp	outside 5.1pp
openai-b	15%	23%	21%	20%	20%	5.2pp	outside 5.1pp
openai-c	19%	18%	22%	18%	23%	2.8pp	within 5.1pp
openweight-a	11%	26%	26%	18%	19%	8.8pp	outside 5.1pp
openweight-b	17%	22%	19%	24%	19%	4.3pp	within 5.1pp
openweight-c	12%	23%	19%	26%	20%	8.4pp	outside 5.1pp
openweight-d	19%	20%	25%	18%	18%	4.8pp	within 5.1pp
openweight-e	17%	17%	24%	22%	20%	3.6pp	within 5.1pp
openweight-f	23%	17%	19%	20%	21%	3.4pp	within 5.1pp
openweight-g	14%	23%	28%	16%	19%	8.0pp	outside 5.1pp
simulant-a	22%	16%	24%	19%	20%	4.4pp	within 5.1pp
xai-a	20%	12%	20%	26%	24%	8.4pp	outside 5.1pp

Under no position bias each slot tends to 20% (n ≈ 250 answered trials, so sampling alone moves shares by up to ±5.1pp). Descriptive pre-series; the registered slot-effect test runs at validation (§6.2.3).

13Behavioral telemetry

verbosity and latency are behavior too

Seat	Median tokens · answer	Median tokens · refusal	Median latency	Trials
anthropic-a	237	·	4756 ms	250
anthropic-b	334	·	7866 ms	250
anthropic-c	224	·	3632 ms	250
anthropic-d	163	·	2818 ms	250
anthropic-e	377	·	7462 ms	250
deepseek-a	571	·	9546 ms	250
deepseek-b	607	·	4295 ms	250
google-b	120	·	0 ms	250
openai-a	86	·	3208 ms	250
openai-b	80	·	2280 ms	250
openai-c	126	·	2852 ms	250
openweight-a	100	·	2922 ms	250
openweight-b	595	1094	13654 ms	250
openweight-c	205	·	1772 ms	250
openweight-d	76	·	2255 ms	250
openweight-e	79	·	2756 ms	250
openweight-f	1541	·	12618 ms	250
openweight-g	115	·	6102 ms	250
simulant-a	12	·	1 ms	250
xai-a	76	·	12298 ms	250

Reversed-keying check

acquiescence seam · reversed items: AJ1, AJ2, AJ3, AJ4, AJ5, AJ6, B2, CAL-02, CAL-03, CL1, CL2, CL3, CL4, CL5, CL6, DO1, DO2, DO3, DO4, DO5, DO6, MG1, MG2, MG3, MG4, MG5, MG6, SF1, SF2, SF3, SF4, SF5, SF6, SI1, SI2, SI3, SI4, SI5, SI6, SV1, SV2, SV3, SV4, SV5, SV6, WE1, WE2, WE3, WE4, WE5, WE6

Seat	Mean · reversed-keyed	Mean · standard	n (rev / std)
anthropic-a	2.78	·	50 / 0
anthropic-b	2.86	·	50 / 0
anthropic-c	2.72	·	50 / 0
anthropic-d	2.91	·	50 / 0
anthropic-e	2.79	·	50 / 0
deepseek-a	2.85	·	50 / 0
deepseek-b	3.03	·	50 / 0
google-b	2.11	·	4 / 0
openai-a	2.91	·	50 / 0
openai-b	2.86	·	50 / 0
openai-c	2.86	·	50 / 0
openweight-a	2.82	·	50 / 0
openweight-b	2.81	·	50 / 0
openweight-c	2.88	·	50 / 0
openweight-d	2.80	·	50 / 0
openweight-e	2.79	·	50 / 0
openweight-f	2.85	·	50 / 0
openweight-g	2.76	·	50 / 0
simulant-a	3.08	·	50 / 0
xai-a	2.77	·	50 / 0

With one reversed item in the smoke pool this is a seam, not a finding. The 48-item core pool balances keying per domain (§2.3 r5), and this table becomes the acquiescence detector.

14The researcher's shelf

what unlocks at each stage of the series, and the section that governs it

Every figure and table above is reproducible from analytics.json plus the per-run trial records; the permutation seed (20260716) ships in the JSON so the p-values re-derive exactly. The methods here are descriptive and additive (semver-MINOR, §5); the frozen §4 statistics they consume never change. Each has a registered inferential successor:

Analysis	Feeds	Status
Stance matrix, distance map, clustering, dispersion, telemetry	this page	Live
Permutation divergence + BH correction across items	the alert stack's correction · §4	Live, descriptive
Variance decomposition, ICC, split-half reliability	the meter's own error bars · §6.2.1	Live, descriptive; registered at pilot
Minimum detectable effect vs n	alert-floor calibration · §4, §6.2.1	Live, descriptive
Drift vs same-build baseline (BH-FDR across model x item)	change alerts · §4	At week 2 of the series
Test-retest reliability across runs, then the alert floor k·SEM	the meter's own error bars · §6.2.1	Validation pilot
Paraphrase invariance (ICC across phrasings)	item survival · §6.2.2	Validation pilot
Position-bias inference (registered slot-effect test)	shuffle adequacy · §6.2.3	Validation pilot
Domain coherence (within-domain correlation structure)	the 8-vector's validity · §6.2.6	Validation pilot
Contamination sentinel (published vs held-out gap)	memorization defense · §2.4	With the item pool
Backbone & sway matrix (blind stance, peer exposure, movement)	social susceptibility · §3.5	Monthly at v1.0

Nothing on this page cost an additional model call. Depth is the free dividend of measuring once and keeping the record.

Cite this Permalink https://modelometer.com/analytics · Run hash d359989e1adcb12631b37545a515ecb07b83c690ce285381390d91accda6c437 · Retrieved 2026-09-18 · derived + inferential analytics over 5000 trials; descriptive pre-series.