AI Research Scientist in London

OxSci is building a credit rating agency for science: a certification layer that combines AI with expert peer review, so researchers, institutions, and AI developers can assess research quality quickly and at scale. The open scientific question at the heart of this company is yours to own: where does the frontier of AI peer review actually stand? What can AI reviewers catch that human experts miss, what do they still get wrong, and how do you prove it rigorously? The standards we set now, including what "quality" even means and how we demonstrate our AI reviewers are actually good, will define both the company and, we believe, the field.

The tech side is led by OxSci's cofounder, a senior tech lead from one of the world's largest technology companies, with deep experience building and operating systems at global scale. You'd work with both founders day to day, shaping the evaluation methodology and research culture from the ground up.

What you’ll do:

• Own the research agenda on AI-reviewer evaluation.
• Track the frontier (AI-scientist, automated-review, LLM-as-a-judge, and scholarly-NLP literature), position our system against it, and decide what we measure next and why.
• Design meta-evaluations that expose weaknesses, not just measure agreement.
• Build fine-grained, criticism-level evaluations of AI review agents (correctness, factual grounding, significance, sufficiency of evidence, hallucination rate, and venue/journal matching) that reveal where and why they fail, going beyond verdict-matching.
• Run expert-annotation studies at scale.
• Design the protocols, rubrics, inter-annotator agreement, and statistics needed to compare AI and human reviewers credibly, including head-to-head evaluations against other AI review systems, and defend the numbers to a skeptical scientific audience.
• Build a living taxonomy of AI-reviewer failure modes such as subfield blind spots, long-context degradation, over-anchoring, and spurious criticism, and turn each into a regression benchmark that guards against backsliding as models and prompts change.
• Calibrate the combined rating.
• Define quality-scoring rubrics for human review reports and calibrate how expert and AI judgment fuse into a single, defensible rating: the core of what universities and publishers buy from us.
• Translate benchmark findings into concrete improvements to our review agents (retrieval, context engineering, orchestration, model choice) and prove the gains with the same rigor you used to find the gaps.

What we’re looking for:

• A PhD (or near completion) in CS, ML, NLP, or a related field, or an equivalent research track record.
• You’re likely already working on LLM evaluation, LLM-as-a-judge, AI for science, automated peer review, or a nearby frontier.
• A fascination with the boundary between AI and human reviewers.
• A track record of rigorous evaluation of ML/LLM systems: benchmark or eval-framework design, evaluator/judge models, expert-annotation study design, hallucination and factuality measurement, uncertainty quantification, or RAG evaluation.
• Fluency in evaluation methodology and statistics: sampling, inter-annotator agreement, significance testing, and the discipline to distinguish a real effect from a lucky prompt.
• Strong Python and hands-on habits.
• Bonus: publications in NLP/ML evaluation or automated peer review; open-source benchmarks or evaluator models the community actually uses; experience with scholarly content at scale.

What we offer:

• Founding seat with meaningful equity and a direct line to the founders.
• Ownership of a genuinely open research question, with encouragement to publish and present the work.
• A standing expert-reviewer network as your annotation infrastructure.
• A proprietary, growing dataset of paired human and AI reviews of real submissions.
• The rare chance to define the standard by which AI reviewers themselves are judged.
• Genuinely competitive pay; equity discussed openly.
• Generous LLM token budget for your daily work.
• Flexible working hours; fast personal growth with broad ownership from day one.