AI Research Scientist

Overview Ox Sci is building a credit rating agency for science: a certification layer that combines AI with expert peer review, so researchers, institutions, and AI developers can assess research quality quickly and at scale.

Webre early, focused, and well-resourced.

The open scientific question at the heart of this company is yours to own: where does the frontier of AI peer review actually stand?

What can AI reviewers catch that human experts miss, what do they still get wrong, and how do you prove it rigorously?

The standards we set now, including what \"quality\" even means and how we demonstrate our AI reviewers are actually good, will define both the company and, we believe, the field.

The tech side is led by Ox Sci's cofounder, a senior tech lead from one of the world\'s largest technology companies, with deep experience building and operating systems at global scale.

You\'d work with both founders day to day, shaping the evaluation methodology and research culture from the ground up.

What you\'ll do Own the research agenda on AI-reviewer evaluation.

Track the frontier (AI-scientist, automated-review, LLM-as-a-judge, and scholarly-NLP literature), position our system against it, and decide what we measure next and why.

Design meta-evaluations that expose weaknesses, not just measure agreement.

Build fine-grained, criticism-level evaluations of AI review agents (correctness, factual grounding, significance, sufficiency of evidence, hallucination rate, and venue/journal matching) that reveal where and why they fail, going beyond verdict-matching.

Run expert-annotation studies at scale.

Design the protocols, rubrics, inter-annotator agreement, and statistics needed to compare AI and human reviewers credibly, including head-to-head evaluations against other AI review systems, and defend the numbers to a skeptical scientific audience.

Build a living taxonomy of AI-reviewer failure modes such as subfield blind spots, long-context degradation, over-anchoring, and spurious criticism, and turn each into a regression benchmark that guards against backsliding as models and prompts change.

Calibrate the combined rating.

Define quality-scoring rubrics for human review reports and calibrate how expert and AI judgment fuse into a single, defensible rating: the core of what universities and publishers buy from us.

Close the loop.

Translate benchmark findings into concrete improvements to our review agents (retrieval, context engineering, orchestration, model choice) and prove the gains with the same rigor you used to find the gaps.

What we\'re looking for A Ph D (or near completion) in CS, ML, NLP, or a related field , or an equivalent research track record.

You\'re likely already working on LLM evaluation, LLM-as-a-judge, AI for science, automated peer review, or a nearby frontier.

A fascination with the boundary between AI and human reviewers.

What each catches that the other misses, and conviction that mapping it rigorously is how trustworthy peer review gets built.

A track record of rigorous evaluation of ML/LLM systems: benchmark or eval-framework design, evaluator/judge models, expert-annotation study design, hallucination and factuality measurement, uncertainty quantification, or RAG evaluation.

Bonus if you\'ve built evaluations that score individual criticisms rather than just verdicts.

Fluency in evaluation methodology and statistics: sampling, inter-annotator agreement, significance testing, and the discipline to distinguish a real effect from a lucky prompt.

Strong Python and hands-on habits.

You build the eval harnesses and pipelines yourself, not just spec them, with enough LLM-application fluency (RAG, tool calling, orchestration) to turn a finding into a shipped improvement.

Bonus: publications in NLP/ML evaluation or automated peer review; open-source benchmarks or evaluator models the community actually uses; experience with scholarly content at scale.

What we offer Founding seat with meaningful equity and a direct line to the founders Ownership of a genuinely open research question, with encouragement to publish and present the work A standing expert-reviewer network as your annotation infrastructure A proprietary, growing dataset of paired human and AI reviews of real submissions The rare chance to define the standard by which AI reviewers themselves are judged Genuinely competitive pay; equity discussed openly Generous LLM token budget for your daily work Flexible working hours; fast personal growth with broad ownership from day one #J-18808-Ljbffr