referee: a skill for refereeing mathematics papers

Ask a language model to referee a paper “without mercy” and you will get exactly what you asked for, which is the problem. The report comes back snarky and inflated: matters of taste presented as defects, every “clearly” treated as a crime, and — the part that actually hurts — a handful of confident accusations about gaps that are not there, because the hypothesis the proof needs was stated twelve pages earlier and the reviewer never saw it. A report with three false accusations loses its authority over the other thirty findings. The author is entitled to discard all of them, and usually does.

referee is a Claude Code plugin with a single skill, paper-referee, that referees a pure-mathematics paper in English, roughly 10 to 60 pages, the way a hard but fair human referee does: the venue’s own rules first, then the proofs, then a report in which every major point is anchored to a line of the manuscript. As with lit-tools, this post is less about what it does than about why it does it that way.

The one design rule

Severity is structural, not tonal.

Tone is free and worth nothing. What makes a review hard to argue with is not its adjectives but the fact that each of its complaints points at a specific place where the paper breaks. So the whole skill is organised around making every finding carry that kind of evidence, and around refusing everything that does not:

• No finding without a failure scenario. Every entry names the concrete circumstance in which the paper is wrong or incomplete: a counterexample, a degenerate instance, a case the argument does not cover, or the exact inferential step with no warrant. “This is unclear” and “I would have preferred” are not major points.
• Burden of proof on the paper. The default verdict on every step is not established. The referee does not ask “can I see anything wrong with this?” but “can I reconstruct this?”. A step that is merely plausible has not been proved, whether or not the statement is true.
• Steelman before you strike. One honest attempt to fill each gap before reporting it. What survives a repair attempt is a finding the author cannot deflect; the attempts that succeed never become findings, which is where most false accusations would otherwise have come from.
• Every quote is re-checked against the source before it reaches the report.
• The recommendation is derived, by a rule over the findings table, so it cannot drift with mood.

The register follows: hard on substance, flat on surface. No praise as cushioning, no “the authors may wish to consider”, no softening a severity to be kind — and equally no sarcasm, no rhetorical questions, no remarks about the authors rather than the paper. A report that reads as hostile gets discounted by an editor as animus, and the real findings go down with the unpleasant ones.

The pipeline

Everything a review produces lives in one directory, review/<slug>/, and the skill runs in phases, each of which writes a durable artefact:

Phase 0 — intake, and the map instead of the territory

python3 ingest.py --tex paper/main.tex --pdf paper/main.pdf
python3 skeleton.py --review review/<slug>

LaTeX source, always, if it exists. From a PDF the displayed mathematics comes out mangled and environment boundaries are guesswork; a PDF-only review is still worth doing, but the degradation is recorded, every proof-level finding drops in confidence, and the report says so rather than absorbing it. The PDF is passed anyway, because it is the only way to count pages.

skeleton.py produces the file the orchestrating model actually reads in place of the paper: sections, every theorem-like environment with its span and its attached proof, the dependency graph between them, and a load-bearing order. Results announced in the introduction come first, marked ★ — that is the author’s own claim about what the contribution is — and then everything else by in-degree: how many later results collapse if this one is wrong. A lemma with in-degree 6 earns the strong model; a remark cited by nobody does not.

Alongside it, hypotheses.tsv lists, per statement, vocabulary that occurs in the hypotheses and never in the proof. These are candidates for an unused hypothesis, not verdicts: a hypothesis can be used through a lemma without its name ever appearing.

Phase 0.5 — the venue’s rules, before any mathematics

Dump the call for papers or the author instructions into a text file, venue-rules.txt by convention, and it is compiled into a checklist in which every rule carries the venue’s own sentence verbatim. That is what makes literal enforcement possible: “violates §3” is arguable; “violates §3: ‘Submissions must not exceed 15 pages excluding references’” is not.

python3 compliance.py --template          # the schema
python3 compliance.py --review review/<slug>

The mechanical checks cover page limits (including the frequent excluding references variant, which the rule’s own quote decides), anonymity — \author, \thanks, acknowledgements, funding statements, self-identifying back-references — required sections, document class and its options, abstract length, appendices, required packages, and arbitrary regexes over the source. Three rules govern the compilation. Rules that need judgement, like scope or significance, are classified as such and answered explicitly in the report; they are never dressed up as automated checks, because a false PASS on scope is worse than an open question. Rules the venue did not write down are not violations. And a mechanical rule with no implemented check returns unknown — a visible gap — rather than being dropped from the list, which would be an invisible one.

A mechanical failure is a desk reject, and the audit does not happen. Reporting on the content of a paper that will not be considered wastes the referee’s time and misleads the author about where they actually stand.

Phase 1 — the intro/theorem contract

Write down what the abstract and introduction promise; write next to it what the numbered statements actually say; compare them literally, word by word. The gap between the two is its own severity class, overclaim, and it is where a large share of real referee findings live: a hypothesis present in the theorem and absent from the advertisement, a conclusion stated more strongly in the introduction than in the body, a “we show that” whose theorem shows something weaker.

This phase also fixes the audit budget: the ★ results get audited in depth whatever else is skipped.

Phase 2 — the audit, in two stages

The orchestrator never reads the paper. It reads the skeleton, the claims file and the mechanical outputs; full text enters only subagent contexts, one slice each. This is the same reason as in lit-tools: the binding constraint is context, not capability, and a window filled with forty pages of proofs degrades exactly when the synthesis matters.

Stage A is a structural sweep over everything, with a cheap model, one subagent per section. It does not judge mathematics. Per statement it asks: is the proof’s shape complete — all cases, a base case, both inclusions, both directions of an iff; are the invoked results in scope at that point; does the proof prove the statement that was announced; where does it assert instead of argue. Its output is a list of suspicions, not findings.

Stage B is the deep audit, with the strong model and never a small one, on every ★ result, everything with in-degree ≥ 2, and everything Stage A flagged. Each subagent receives the statement, its proof, the statements only of its dependencies, the relevant notation, and a checklist of where pure mathematics actually breaks. It must reconstruct the argument independently before comparing it with the paper’s, and it must attempt a repair for every gap it wants to report. Audits run in dependency layers, leaves first, so that an audit inheriting a broken lemma knows it is broken instead of rediscovering it.

The checklist is the part of the plugin where domain knowledge lives, and it reads like what it is — a list of questions to answer, most of which produce nothing:

• Degenerate instances. Instantiate at the empty set, the zero module, the one-object category, the point, $n = 0$. The statement is fine, false, or vacuous — and a vacuous case in a chain of lemmas often means the induction has no base.
• Hypothesis load. Delete each hypothesis and try to prove the statement anyway. If it still goes through, the hypothesis is unused (say so: it is a service) or the proof uses it somewhere unmarked. If it does not, find the counterexample the hypothesis excludes — the best possible evidence that the proof is honest.
• Quantifier order and uniformity, WLOGs as unpaid proof obligations, “similarly” applied to the asymmetric case.
• For category theory specifically: size, and above all naturality — an isomorphism constructed objectwise and later used as if natural, which is the single most common unstated gap in the subject — functoriality asserted on morphisms, coherence declared routine, adjunctions without triangle identities, 2-cells where there should be equalities or vice versa.

Whatever the budget did not reach goes, by name, into the report’s “not covered” section. A silently partial audit is the one dishonest outcome.

Phase 3 — what must not depend on attention

python3 hygiene.py --review review/<slug>

A handwaving census (obvious, clearly, routine, well known, left to the reader, …), weighted by whether each occurrence sits inside a proof; repeated phrasing by shingling; dangling \ref and \cite, unused labels, uncited bibliography entries, unused macros, vague forward references. All of it arrives already anchored, at zero model tokens, and it is done by a script rather than by eye because it is precisely the part of a referee report that must not depend on whether the reader was still paying attention on page 41. Its completeness is also what tells an author that the manuscript was actually read.

On top of it, the global passes that no per-statement audit can see: circularity in the dependency graph, notation reused for two things, a definition given twice inequivalently, a result that is a special case of another result in the same paper.

Phase 4 — novelty, or an honest admission

If a lit-tools corpus covers the area, each ★ statement is handed to lit-claim-search, which answers who already said it, said less, said more, or said the opposite. If not, the report declares novelty unassessed. It never implies a literature check that did not happen.

Phase 5 — the gate

Every finding goes into a single table,

id · severity · category · location · quote · failure · repair · confidence · verified · note

and then:

python3 verify.py --review review/<slug>

verify.py searches for each quote in a whitespace- and punctuation-free mirror of the manuscript, and checks that it sits where the finding says it does. A quote that exists elsewhere is marked moved and its location corrected. A quote that cannot be found is dropped, not rewritten to fit. A finding with no failure scenario is not a finding. This script exists because the expensive failure of a hard review is not the missed flaw; it is the confident accusation built on a sentence the paper does not contain.

Phase 6 — two reports, and a recommendation computed from a table

Severity classes are about consequence, never about irritation:

unsupported deserves a word: it is the category for what an author experiences as unfairness and a referee recognises as the commonest real defect — the paper knows something it never told the reader, and the proof leans on it.

The recommendation is then derived, not felt:

1. any verified fatal → reject;
2. else any gap or unsupported on a ★ result or on a node of in-degree ≥ 2 → major revision;
3. else any overclaim → major revision, because the fix is a rewrite of the framing, not a typo pass;
4. else any gap or unsupported anywhere → minor revision;
5. else → accept, with the hygiene list attached.

REPORT.md goes to the author: a summary of the paper in the referee’s own words (if it cannot be written, the paper was not read well enough to review, and the review stops there), an assessment, the compliance answers rule by rule, major and minor points each with location, verbatim quote, failure and fix, and a section headed what this report does not cover. EDITOR.md stays confidential: the recommendation and which clause fired, one sentence of reason, and — labelled as an opinion — whether the contribution, if correct, is worth the venue. Whether a result is interesting is the editor’s question; it never appears in the author-facing report disguised as a defect.

Re-review

Given the revised version, the previous findings table and the authors’ response letter, the report opens with one row per previous finding — addressed, partially, deflected, ignored — each checked against the actual diff. A response letter says where to look. It is never evidence that anything was fixed.

On the other side of the desk

The same pipeline run by an author on their own manuscript is simply a self-check. The compliance gate catches what gets papers desk-rejected, the hygiene pass catches what referees notice first, and the intro/theorem contract catches the overclaim you did not know you were making because you have been reading your own introduction for eight months. That is the use I expect to be most common: being torn apart by something that is obliged to tell you exactly where, before a human does it without that obligation.

What it will not do

Report aesthetic preference as a defect. Judge interest in the author-facing report. Assert novelty without a corpus. Soften a severity to be kind, or sharpen a sentence to sound severe. The same principle as in lit-tools runs through all of it: the referee is allowed to be ignorant — about a cited result it did not consult, about a proof it audited only structurally, about the literature — but never allowed to present ignorance as a verdict.

referee needs Python 3.11+ and no third-party packages; pdftotext and pdfinfo (poppler-utils) only for PDF sources and page counting. It lives in the same marketplace as lit-tools: claude plugin marketplace add tetrapharmakon/skills, then claude plugin install referee@tetrapharmakon, or the same two steps from /plugin inside Claude Code.