Does the classifier get better as you label?

Every figure below is measured on the reserved held-out slice only, carries the n it was measured on, and is suppressed to an interval below n=30. Nothing on this page is simulated, illustrative, or carried over from a previous run.

span learner - held-out accuracy as the owner labelsappend-only eval ledger - 1 run recorded - source=owner - every figure measured on the reserved held-out slice onlyNo learning curve yet - and that is a fact about the labels, not a failure.NEVER MEASURED. This is not 'zero accuracy' and not 'accuracy unknown because something broke'.No run has produced a held-out accuracy figure on owner labels, because there is nothing yet tomeasure one on.Owner judgements recorded so far: 3Of those, landing in the reserved evaluation slice: 0A figure needs labels on the EVALUATION side. Training labels teach the model; only held-outlabels can score it.First interval (no point yet) needs 1 more held-out label - roughly 7 judgements in total at themeasured 14.9% held-out rate.First point estimate needs 30 more held-out labels - roughly 201 judgements in total. Below n=30the point stays suppressed and only the interval is drawn.That total is an ESTIMATE, not a promise: held-out membership is sha256(seed|span_id), so whichof your judgements land on the evaluation side is decided by the span, not by a coin.1 run already recorded in the append-only eval ledger, all with n=0 on the owner side. Themachinery is live; it is waiting on labels.▶ Label the next batch the classifier is least sure about: we leave

Where you stand

owner judgements recorded3
of those, in the held-out evaluation slice0
held-out labels still needed for the first interval1
held-out labels still needed for a point estimate30
estimated total judgements for the first interval7
estimated total judgements for the first point201
training runs recorded in the append-only ledger1

an ESTIMATE from the observed held-out fraction, not a promise. Held-out membership is sha256(seed|span_id), a deterministic function of which spans the review tool happens to show. Judge 100 spans that all hash to the training side and the eval slice stays at zero.

Judge this next

Judge held-out spans under the trigger 'we leave' first: 4 of the 25 most-uncertain held-out spans fire on it, so one batch there converts the most judgements into evaluation ground truth. The single most uncertain span is 0943a8d721277f31 (p=0.5000, i.e. the model is at 50.0% -- a coin flip).

Open the review tool on this batch

Recommendation from next-labels.1787526028.json

Rerun it

After you label more, trigger a retrain. It runs contract → extract → train → evaluate and appends a new point; it never rewrites an old one.

curl -s -X POST http://127.0.0.1:8266/api/orion/learn/retrain -H "X-Orion-Assert-Token: $TOKEN" -d '{"labels":"owner"}'

Calling it twice is safe: the second call returns HTTP 409 with the running run's id and starts nothing.

Last run: {"run_epoch": 1787525043, "state": "done", "step": "done", "runtime_s": 986.1, "contract_status": "fail", "label_source_used": "proxy"}

Span by span

The curve above is an aggregate. To see what the classifier said about one span — its probability, its inertia type and runner-up, the stored label it is being compared against, and whether they agree — open the per-span view. It defaults to the confident disagreements and breaks the held-out scores down by area.