There are two classifiers, they answer different questions, and they are
scored differently. Every figure on this page is read live from
/api/orion/learn/eval at the moment you loaded it; nothing here is
baked into the page. reading live values…
Read this before any number below. The labels these classifiers are trained
and scored on are machine labels, not human judgements
(label_source). A span counts as a constraint if the existing
pipeline marked it classified != NONE. So every accuracy, precision and
F1 on this page measures agreement between two machines — it is not
evidence that either is right.
Human judgements recorded so far: —. Until that number is substantial, treat everything here as a comparison between methods on identical data, never as a measure of truth.
| span learner | type learner | |
|---|---|---|
| question | Does this sentence state a constraint the authors hit? | Which of the seven inertia types is it? |
| shape | binary | 7-way |
| headline metric | F1, because the classes are uneven | macro-recall, because one class is 96× another |
| trained on | — | — |
| scored on | — | — |
| features | — | — |
They are frequently conflated. A paper can be correctly identified as stating a constraint and then given the wrong type; those are two separate failures and the review tool asks about them at two separate stages.
A span is reserved for evaluation by a pure function of its own id, so the same span always lands on the same side, on every run, on every machine:
Seed —, fraction —. The learning lane recomputes this itself rather than trusting the value it is handed, and asserts it at every fit. Nothing that trained the model is ever scored on.
Why it matters: if the split were random per run, a good score could be luck, and re-running would quietly change the answer. Because membership is a hash of the span id, the evaluation set is identical today and next month.
The model emits a probability. Turning that into yes or no needs a threshold, and the obvious choice — 0.5 — is wrong here.
The model was calibrated with isotonic regression fitted on a 20% slice of the training pool; the held-out set is never used to calibrate. But the stored model carries the classifier without that calibrator. Scoring at a naive 0.5 gives 0.7911 accuracy where the published figure is 0.8066 — a panel that would silently disagree with its own evaluation by 1.5 points.
Because isotonic regression is monotone, the published operating point can be recovered as a rank on the raw scores. That threshold is —, and it is verified by reproducing the published confusion matrix exactly before it is used. If that check ever fails, the panels fall back to 0.5 loudly rather than quietly.
Figure 1 — what the calibrator did, on the held-out spans
drawing from the live read…
Figure 2 — what any other threshold would have cost
drawing from the live read…
An unweighted 7-way model reaches 0.9123 accuracy and only 0.3507 macro-recall. It gets that accuracy by answering methodological almost always, which is right most of the time and useless. Macro-recall averages the per-class recall, so ignoring a rare class costs as much as ignoring a common one.
Every model here is scored against the trivial alternatives on the identical held-out set:
| span learner | F1 | type learner | macro-recall |
|---|---|---|---|
| the learner | — | text only | — |
| admit everything | — | text + vocabulary prior | — |
| the phrase vocabulary | — | vocabulary prior alone | — |
| always say methodological | — |
Figure 3 — the span learner beside its trivial alternatives
drawing from the live read…
Figure 4 — macro-recall and accuracy, pulling in opposite directions
drawing from the live read…
Figure 5 — where the seven-way errors actually go
drawing from the live read…
Two things fall out of that table and both matter more than the headline. The phrase vocabulary scores almost exactly the same as admitting everything. And adding the vocabulary to the text learner changes its macro-recall by less than one part in a thousand.
Measured directly: trigger identity carries 0.6291 bits about the outcome; the 4-way cluster label alone carries 0.5830 bits — 92.68% of everything the 1,128-phrase vocabulary knows. The remaining 741-way distinction between individual triggers is worth 0.046 bits. The vocabulary is not uninformative; it is redundant with the text it matched.
Figure 6 — the vocabulary against a four-way cluster label
Every figure in the panels carries the denominator it was computed over, and intervals are Wilson score intervals, not the normal approximation, because the normal approximation misbehaves at small n and near 0 or 1 — exactly where these classes sit.
Class support is severely uneven, and that governs which figures exist at all:
| inertia type | training spans | reported? |
|---|---|---|
| reading… | ||
Figure 7 — class support on a logarithmic axis
drawing from the live read…
The per-area panel applies an explicit support policy: a point estimate at n ≥ 100, interval only at 30 ≤ n < 100, and refused below n = 30. On the current data that refuses 106 of 156 slices — 75 outright, 31 point-suppressed — each named with its n rather than silently dropped. Of 136 arXiv categories only 32 clear the bar.
The corpus was extracted with pdftotext -layout, which interleaves the two
columns of a two-column paper. 63.5% of spans carry extraction damage (n=400).
The classifier does not know this. Sorted by confidence, its top disagreements include
spans like ⊓ ⊔ ⊓ ⊔ We cannot
scored as a real constraint at p = 1.00000.
Re-extraction fixes recall (+4.9% documents, +10.2% on two-column papers) but not precision — 0.5784 → 0.5806, and 88.2% of adjudicated false positives survive it. Those are meaning errors, not extraction errors.
Sampling the model three times per span and keeping only unanimous answers does not improve accuracy: unanimous 3/3 scored 30.8% against 33.3% for split votes (n=40). Self-consistency carries no signal here, so a “keep only the confident ones” gate would discard most of the work and buy nothing.
DARPA’s SCORE programme spent seven years and 3,900 claims on automated replication prediction. Humans reached 76–78%; none of three machine methods was consistently effective. ORION therefore reports what a paper says stopped its authors and what it would take to re-run it — never a prediction that the result is true.
The binding constraint is not compute. One judgement costs about 0.1 seconds of GPU; a corpus-wide pass is roughly 25 GPU-hours. The binding constraint is labels.
POST /api/orion/learn/retrain,
measured at about 16 minutes end to end, appending a new evaluation record each time so
the curve extends rather than being overwritten.