Our practical assessment and our managed projects only work if the standards behind them are excellent. The AI Evaluation Quality Lead owns those standards: the assessment tasks and answer keys, the guidance reviewers follow, the quality bar on managed evaluation and annotation projects, and the feedback members receive. You will have worked hands‑on in LLM evaluation, data annotation or search quality and know what good looks like.
What you'll do
- Design and maintain assessment content, rubrics and reviewer guidance so decisions are consistent and defensible
- Set and check quality standards on managed projects: sampling, calibration and feedback loops
- Review AI advisory grading output and decide where it helps and where it misleads
- Write development guidance members can act on after a retry decision
- Work with the Talent Operations Lead on reviewer training and calibration sessions
What we're looking for
- Hands‑on experience in LLM evaluation, data annotation quality, search or ads quality rating, or similar
- You can write a rubric that two reviewers apply the same way
- Analytical and precise; comfortable with sampling and simple metrics
- Right to work in the United Kingdom or Nigeria
Nice to have
- Experience training or calibrating raters at scale
- Familiarity with responsible‑AI evaluation practice
- Experience with multilingual evaluation
What we offer
- Remote‑first with flexible hours across UK and West Africa time zones
- Real ownership of standards that affect how people are assessed
- Equipment provided and a learning budget
#J-18808-Ljbffr
