Journal DOI 10.68956/dyh5-kwte

Editorial Opinion

What Should AI Catch Before an Editor Sends a Review?

A medical journal should evaluate an AI review tool by the important errors it helps editors catch, the extra work it creates, and the confidentiality it preserves.

Neural Horizons

A recommendation is a poor test of review quality

Bastien Le Guellec and Su Hwan Kim describe an experience familiar to anyone who has asked a language model to critique a draft. Fixing one set of objections produces another. Their essay argues for keeping models in an advisory role. Even in that advisory role, a system should earn its place by improving a specific editorial task.

In the authors’ related preprint, four models evaluated 200 articles from 20 journals. Across 800 reviews, 605 recommended major revision or rejection. The researchers also inserted one domain-specific flaw into versions of the papers. Models detected the planted flaw in 17.0% to 60.5% of cases, depending on the model. These are results from a preprint, under its particular sampling and prompting conditions.

The published articles give the study a reference point for an actual editorial outcome. Publication alone does not establish that a paper is free of serious flaws. The study’s sample also contains no rejected submissions, so agreement with the original decisions cannot tell us how well the models distinguish submissions that merit acceptance from those that do not. A useful objection can appear in a review with an overharsh recommendation. A confident recommendation can also come with no useful objection. The planted-flaw exercise gives us a more direct measure of one capability, but it also depends on how detection was scored. The study used one model to insert the flaws and another to grade detection, with partial detections counted as misses. Human adjudication of real editorial discrepancies would test whether the result carries over to practice.

Reference 1Reference 2

Start with a claim an editor can verify

A reasonable first task for a medical journal would be to check whether the abstract, participant-flow diagram, and main results table report consistent denominators. In a diagnostic study, the denominator can legitimately differ between tables because some participants lack an evaluable reference standard. The tool should point to the exact figures and passages and say what appears inconsistent. A human editor with the appropriate expertise decides whether the discrepancy is real and whether it changes interpretation.

Feedback may still be useful even if verdicts are unreliable. Liang and colleagues found that many researchers in their user study regarded model feedback as helpful. The study’s feedback overlap and usefulness ratings answer different questions from whether a system detects consequential defects or saves editorial time. A journal evaluating deployment should measure those outcomes directly.

Reference 3

Test the editorial workflow, including when to stop

Start with publicly available manuscripts. Have two independent subject or methods editors label a sample of authentic discrepancies and legitimate variations, resolving disagreements before evaluating the tool. Compare editors working with their usual process against editors who receive model flags after an initial independent read. Report the share of consequential defects caught, the share of flags that were valid and useful, the time editors spent adjudicating them, and the additional revision requests authors would receive. Report missed defects and false flags by task. Repeat the evaluation after a material model or prompt change.

Keep model output internal until a human confirms a material issue against the cited passages. Log the model version, task instructions, and editorial disposition. After a revision, check the issues already agreed with the authors. Escalate a new issue only when an editor independently determines that it materially affects interpretation, validity, or research integrity.

Confidentiality is a prerequisite. ICMJE guidance cautions against sending submitted manuscripts to AI systems where confidentiality cannot be assured without explicit author permission. A test using public papers can start without disclosing unpublished submissions. Any later use should follow the journal’s confidentiality policy and applicable permissions.

Reference 4

For a medical journal, the pilot earns a second round only if it helps editors catch consequential mistakes at an acceptable cost in time and false alarms. Editors remain responsible for deciding what authors are asked to change.

References

  1. Down the Rabbit Hole of LLM Peer Review

    Le Guellec B, Kim SH. Radiology: Artificial Intelligence Substack. November 11, 2026.

  2. Major Revision, Minor Accountability: LLMs as Reviewers of Published Articles in Top-Tier Journals

    Le Guellec B, Kim SH, Bentegeac R, Hamroun A. arXiv preprint. October 22, 2026.

  3. Can Large Language Models Provide Useful Feedback on Research Papers? A Large-Scale Empirical Analysis

    Liang W, et al. NEJM AI. 2024.

  4. Use of Artificial Intelligence in Publishing

    International Committee of Medical Journal Editors. Recommendations.