ESRC Trusted AI Project Post-Doctoral Researcher in Newcastle upon Tyne

As a Research Associate, you will take a leading role in the project Trustworthy AI for Peer Review: Developing and Validating Composite Research Quality Indicators. Your work will focus on constructing high-quality REF-aligned corpora of research outputs, using and extending the full-text cache and metadata pipelines developed through our software: Macroscope. You will collect, clean, link, and document text and bibliographic data so that it can support rigorous testing of AI-assisted research assessment.

You will be responsible for developing reproducible data workflows that link REF2021 submissions, Units of Assessment, outcome profiles, institutional metadata, bibliographic records, abstracts, and, where licensing permits, full-text content. This will involve working with sources such as OpenAlex, Crossref, Unpaywall, Scopus and REF data, designing robust extraction and matching methods, maintaining provenance and quality-control records, and producing well-structured datasets for downstream analysis.

The role will also contribute to the project’s LLM-based evaluation work, helping to design, run, and analyse experiments that test how large language models perform in REF-style assessment tasks. This will include supporting prompt and model comparison, repeated-run evaluation, assessment of stability and disagreement, and analysis of potential biases or confounding effects.

The post provides an opportunity to work at the frontier of trustworthy AI, research evaluation, scientometrics, and large-scale text infrastructure, with scope to publish, present, and help shape policy-relevant demonstrators. The successful candidate will also have the opportunity to support in the development of Semantic Space, an Academic Intelligence company that supports strategic decision making for research intensive organisations. The role is fixed term for 12 months.

Trustworthy AI for Peer Review: Developing and Validating Composite Research Quality Indicators is a UKRI/ESRC Metascience project investigating how artificial intelligence can support high-stakes research assessment without displacing expert judgement. Research assessment through peer review, journal editorial processes and the Research Excellence Framework (REF) is essential to the research system, but it is increasingly costly, time-consuming and difficult to scale. At the same time, large language models are creating new possibilities for analysing and evaluating research outputs.

The project addresses a central challenge: AI-derived research indicators are often presented as single scores, with limited visibility of uncertainty, disagreement, instability or bias. The project will develop and validate new approaches that make the reliability of AI-augmented assessment explicit and usable by decision-makers. It will combine benchmark datasets, LLM evaluation, scientometric indicators, uncertainty-aware modelling, Trust Cards, and user-centred demonstrators for REF panels, journal editors, publishers and research managers.

Key research areas include:

• Building FAIR, multi-resolution benchmark datasets linking REF submissions, journal peer-review data, books, bibliographic metadata, abstracts and, where appropriate, full-text corpora.
• Developing and testing LLM-based approaches to REF-style and peer-review assessment, including repeated-run evaluation, prompt comparison, model comparison, and calibration against known assessment outcomes.
• Creating indicators that communicate uncertainty, disagreement, stability, bias sensitivity and limits of valid use, rather than reducing complex judgements to opaque single scores.
• Developing transparent governance and communication tools, including Trust Cards, open-source workflows, documentation and demonstrators for responsible use of AI in research evaluation.

The project is delivered by an interdisciplinary consortium led by Loughborough University with Northumbria University, Durham University, Heriot-Watt University, the University of Southampton, the University of Aberdeen and the University of Wolverhampton, bringing together expertise in machine learning, scientometrics, accountable AI, computational reasoning, human-centred design, metascience and research evaluation practice.

The project team brings together researchers working across trustworthy AI, machine learning, scientometrics, human-centred design, computational reasoning, research policy and responsible innovation. The wider consortium includes specialists in uncertainty-aware AI, peer-review analysis, accountable AI, argumentation, stakeholder engagement and research evaluation.

At Northumbria University, the role will be based within Professor Martyn Dade-Robertson’s research environment, building on existing work on Macroscope: a software and data pipeline for mapping, analysing and interpreting large corpora of academic research. The Northumbria contribution focuses on the empirical data backbone of the project, including corpus construction, full-text and metadata processing, REF-aligned benchmarking, and interpretable demonstrators that help experts understand complex research landscapes.

We are looking for a highly motivated postdoctoral Research Associate with strong skills in data handling, computational text analysis and reproducible research. You will have experience of programming for data-intensive research, ideally using Python and associated tools for APIs, web data extraction, text processing, databases, data cleaning, version control and workflow documentation. Experience with NLP, large language models, bibliometrics, scientometrics, research information systems, OpenAlex/Crossref/Unpaywall, REF data, or full-text extraction pipelines would be especially valuable. You do not need prior experience of REF assessment, but you should be interested in how data and AI can be used responsibly in high-stakes research evaluation.

Further information about the requirements of the role is available in the person specification. If you would like an informal discussion about the role, please contact Prof Martyn Dade-Robertson. To apply for this vacancy, please click 'Apply Now'. Your application should include a covering letter and a CV.

Join Northumbria University, a research-intensive institution unlocking potential and changing lives locally and globally. Named Times Higher Education's University of the Year in 2022 and Modern University of the Year in 2025, we rank top 25 in the UK for research power. Discover more about us. With over 37,000 students from 140+ countries, we offer world-leading research, award-winning partnerships, and an outstanding student experience. We empower our exceptional staff, promoting a positive work-life balance and offering great benefits, including excellent pension schemes, flexible working, generous holiday entitlement and more.

Our Northumbria Values, co-created by our team, define who we are: Academic Excellence, Innovation, Inclusivity, Collaboration, and Ambition. Our Behaviours shape our work culture: We listen and learn, support one another, respect everyone, trust each other, and are bold. Based in Newcastle upon Tyne and London, we are an on-campus organisation and offer flexible hours and location where the role allows. We pride ourselves on diversity and inclusivity, holding numerous awards for gender and race equality, disability confidence, and research excellence. We also hold the HR Excellence in Research award for implementing the concordat supporting the career Development of Researchers and are members of the Euraxess initiative to deliver information and support to professional researchers. The University has implemented a range of flexible working arrangements, and we are happy to explore candidate requirements as part of the recruitment process. Apply now and join a community committed to transforming society and the economy.