You are the only source of ground truth in this system. Everything else — the vocabulary, the classifier, the assessments — is machinery grading its own homework. This page explains exactly what each question is asking, what the screen is showing you, and where the hard cases are.
If you read one section, read §3. The single distinction that matters most is “chose to stop” versus “was stopped”. Getting that right is worth more than everything else on this page, because it is the failure the machine cannot fix by itself.
Precision is the bottleneck, not recall: re-extracting the entire corpus moved precision 0.5784 → 0.5806, and 88.2% of false positives survived it. They are meaning errors. Only a human fixes those.
A sentence goes through two independent judgements. A span can be a genuine constraint and be given the wrong type; those are two different mistakes, so you are never asked to settle both at once.
| stage | the question | your choices |
|---|---|---|
| span | Did the authors say something stopped them? | 3 YES 1 NO 2 SKIP |
| inertia | Is the type proposed for it the right one of the seven? | 3 YES 1 NO 4 NOT A LIMIT 2 SKIP |
The digits are stable across both stages: 3 is always affirmative, 1 is always negative, 2 is always skip. 4 exists only on the type stage and means the span should never have been admitted at all — it is the escape hatch when the first stage got it wrong.
| choice | means | use it when |
|---|---|---|
| 3 YES | these authors say something prevented them | the sentence reports a limit they ran into, in their own voice |
| 1 NO | a citation, an aspiration, a choice of scope — not something that stopped them | the limit belongs to someone else’s work, is hypothetical, or is a decision they made |
| 2 SKIP | the sentence does not settle it | genuinely ambiguous, or the text is too damaged to read — say which, in the note |
| 4 NOT A LIMIT | should never have been admitted at all | on the type stage, when the span is not a constraint in the first place |
SKIP without a reason is the least useful thing
you can do. Of the 30 labels cleared earlier, 4 were cannot_tell with
nothing recorded about what was missing — which is exactly the signal that
would tell us how to improve the tool. If you skip, write why: ambiguous, damaged text,
missing context, needs the full paper.
Almost every trigger phrase in the vocabulary states a constraint exists: they do so 95.2% of the time. But only 39.3% survive adjudication. The gap is almost entirely one thing: authors constantly write sentences that sound like limits but are actually scoping decisions.
| phrase | spans it caught | survives | reading |
|---|---|---|---|
beyond the scope | 64,071 | 0.010 | chose to stop |
we leave | 39,958 | 0.076 | chose to stop |
future work | — | 0.001 | chose to stop |
restrict our | 22,487 docs | 0.075 | chose to stop |
cannot afford | — | 0.983 | was stopped |
computationally expensive | 3,548 | 0.963 | was stopped |
prohibitively expensive | — | 0.909 | was stopped |
The two clusters differ by 23× in yield, and the deferral cluster consumes about 20.7% of every span in the corpus. So:
The useful test: if someone handed them unlimited time, money and hardware tomorrow, would the sentence stop being true? If yes, it is a real constraint. If they would still have made the same choice, it is scoping.
On the type stage you are shown one proposed type and asked whether it fits. Each is defined the same way here as on the public Evidence tab.
| type | what it means | training spans |
|---|---|---|
| reading live… | ||
Three of the seven are thinly evidenced.
When the tool proposes paradigm, credibility or
disciplinary, it is generalising from a very small number of examples, and the
screen says so. Be correspondingly willing to answer 1 NO.
methodological has the opposite problem — it is the largest class and
the least useful, because it mixes a genuine absence of technique with
“beyond the scope of this paper”.
Above the sentence is everything known about the paper it came from: title, year, field, estate; the three machine verdicts named separately so you can see when they disagree; the composite score; and the paper’s other spans with their types, with the one you are judging marked ▸ THIS ONE.
Fields carry one of three states, and they are not the same thing:
The card on screen is the exact object hashed into your label as
card_sha256. What you saw and what gets recorded are provably the same thing,
and a label whose card fails verification is refused rather than stored.
You are told which phrase caught this sentence, how many spans it has caught, and how those were classified — so you know whether you are looking at a reliable trigger or a noisy one before you rule on it. It is labelled as the machine grading its own work, because that is what it is.
The corpus was extracted with a setting that interleaves the two columns of a two-column paper. 63.5% of spans carry damage. You will see sentences like:
That is column 1 and column 2 spliced together. The screen flags it — gutter run, hyphen-split word, starts mid-sentence, caps block, citation marker — and attributes it to our pipeline.
Do not judge shredded text. SKIP it and note that the extraction is damaged. If you rule NO on a garbled span, that is recorded as a judgement about the paper, and nobody can later separate our extraction failures from real negatives. A re-extraction is running now that removes most of this.
Judgements recorded so far: —. The learning curve needs held-out labels specifically — training labels alone will never produce a point on it.
we cannot currently sits at a
Wilson lower bound of 0.5987 against a bar of 0.60 — thirteen
thousandths short, carrying 26,062 new documents. A few dozen judgements decide it.beyond the scope and its relatives are
~20.7% of all spans and survive at under 5%. Judging them is mostly confirming a
foregone conclusion.1 negative 2 skip 3 affirmative 4 not a limit (type stage) ? key overlay in the tool.