Find a job 
Featured Companies 
Featured Universities 
For Companies At a Glance
- Tasks: Design and operate AI evaluation and testing for production-grade systems.
- Company: Leading financial services organisation focused on innovative AI solutions.
- Benefits: Competitive pay, flexible working, and opportunities for professional growth.
- Other info: UK-based contract role with potential for significant impact in AI governance.
- Why this job: Join a cutting-edge team to shape the future of AI in finance.
- Qualifications: Experience in UK financial services and strong AI/ML knowledge required.
The predicted salary is between 60000 - 80000 £ per year.
A leading financial services organisation is seeking an AI Evals & Red Teaming Expert to design and operate a robust evaluation and adversarial testing capability for production-grade AI systems. The successful candidate will be responsible for implementing automated adversarial testing within CI/CD pipelines using tools such as AgentDojo, Garak, and Pyrit, with formal release gating to ensure safe and compliant deployment of AI systems.
They will establish and own a comprehensive AI measurement framework, including success rate tracking, uncertainty quantification, and model drift detection. This includes building repeatable evaluation standards that can be applied across all agentic systems within the enterprise. A key aspect of the role involves close collaboration with security and governance stakeholders to map threats to test cases and generate EU AI Act (Article 15) compliance evidence.
The role also includes ownership of the organisation’s AI Bill of Materials (AI-BOM), ensuring supply chain integrity, monitoring model drift, and maintaining signed artefacts across the AI lifecycle. The expert will design and implement testing strategies covering bias detection, hallucination analysis, and memorisation risk assessment, embedding these into a centralised evaluation platform used across all AI systems in production.
Common requirements across all AI engineering roles in the organisation include:
- Strong experience within UK financial services, with working knowledge of DORA, FCA Operational Resilience, and the EU AI Act
- Hands-on experience with AWS Bedrock (including Agents, Knowledge Bases, Guardrails, and model lifecycle management)
- Solid understanding of AI/ML fundamentals, including foundation models, RAG architectures, non-deterministic agent behaviour, and tool-using systems
- Strong knowledge of secure AI practices, including OWASP LLM Top 10, agentic AI threat modelling, and familiarity with the NIST AI Risk Management Framework (AI RMF)
This will be a UK based inside IR35 contract role working via Umbrella Company so you must be resident in the UK to be considered for this role.
AI Evaluation and Teaming Consultant in London employer: Cognitive Group | Part of the Focus Cloud Group
Contact Detail:
Cognitive Group | Part of the Focus Cloud Group Recruiting Team
StudySmarter Expert Advice 🤫
We think this is how you could land AI Evaluation and Teaming Consultant in London
✨Tip Number 1
Network like a pro! Reach out to folks in the financial services sector, especially those working with AI. Attend industry meetups or webinars, and don’t be shy about sliding into DMs on LinkedIn. We all know it’s not just what you know, but who you know!
✨Tip Number 2
Show off your skills! Create a portfolio showcasing your experience with tools like AgentDojo and AWS Bedrock. We recommend building a mini-project that highlights your understanding of adversarial testing and compliance. This will make you stand out when you apply through our website.
✨Tip Number 3
Prepare for interviews by brushing up on key concepts like model drift detection and bias analysis. We suggest doing mock interviews with friends or using online platforms. The more comfortable you are discussing these topics, the better your chances of landing that role!
✨Tip Number 4
Follow up after interviews! A quick thank-you email can go a long way. Mention something specific from your conversation to remind them why you’re the perfect fit. We believe this little gesture can keep you top of mind as they make their decision.
We think you need these skills to ace AI Evaluation and Teaming Consultant in London
Some tips for your application 🫡
Tailor Your CV: Make sure your CV speaks directly to the job description. Highlight your experience with AI evaluation and testing, especially in financial services. We want to see how your skills align with what we're looking for!
Showcase Relevant Projects: Include specific projects where you've implemented adversarial testing or worked with tools like AgentDojo or Pyrit. This gives us a clear picture of your hands-on experience and how you can contribute to our team.
Be Clear and Concise: When writing your application, keep it straightforward. Use bullet points for key achievements and avoid jargon unless it's relevant. We appreciate clarity and want to understand your qualifications quickly!
Apply Through Our Website: We encourage you to apply directly through our website. It’s the best way for us to receive your application and ensures you’re considered for the role. Plus, it makes the process smoother for everyone involved!
How to prepare for a job interview at Cognitive Group | Part of the Focus Cloud Group
✨Know Your AI Fundamentals
Make sure you brush up on your AI/ML fundamentals, especially around foundation models and non-deterministic agent behaviour. Being able to discuss these concepts confidently will show that you have a solid grasp of the technical aspects required for the role.
✨Familiarise Yourself with Compliance Standards
Since the role involves generating compliance evidence for the EU AI Act, it’s crucial to understand its requirements. Be prepared to discuss how you would map threats to test cases and ensure compliance in your previous roles or projects.
✨Showcase Your Hands-On Experience
Highlight any hands-on experience you have with tools like AWS Bedrock, AgentDojo, or Garak. Share specific examples of how you've implemented automated adversarial testing or built evaluation frameworks in your past work.
✨Collaborate and Communicate
This role requires close collaboration with security and governance stakeholders. Be ready to discuss how you’ve worked with cross-functional teams in the past, and demonstrate your ability to communicate complex ideas clearly and effectively.