AI Engineer - Evals

The Team

The AI Team, part of Front Office Technology, builds the AI tools and platforms used across the front office: assistants and agents for investment teams, secure access to firm data and knowledge, and the controls that let AI be used safely in a regulated trading environment.

Role Purpose

We build central capabilities for the whole firm and support teams across the business as they build their own, bringing the most successful into Technology ownership as they grow in importance.

The Team

The AI Team, part of Front Office Technology, builds the AI tools and platforms used across the front office: assistants and agents for investment teams, secure access to firm data and knowledge, and the controls that let AI be used safely in a regulated trading environment.

The Role

As an AI Engineer with a focus on evals, you will help build the AI tools and platforms used across the front office, and the evidence that tells us whether they are accurate, safe and worth the spend.

AI-assisted development is becoming the default for our engineers, and assistants and agents are taking on more work in front office workflows.

Decisions about which models, tools and changes to ship should rest on measurement, not impressions.

You will build on our existing evaluation framework rather than start from scratch, and help make good evaluation practice a normal part of how AI is built across the firm.

The team is small and priorities shift as the firm’s use of AI grows.

You will be expected to pick up other priorities across the team as they arise, so this role will not be dedicated solely to evals.

• Key Responsibilites
• Evolving our evaluation framework
• Extend our evaluation framework to cover new assistants, agents and use cases, improving its design, reliability and ease of use.
• Set the direction for AI evaluation, balancing new capability against keeping the framework simple to adopt.

Education and advocacy

• Educate engineers and the business on why evals matter, through documentation, workshops, pairing and code review.
• Help teams define what good looks like for their use cases and turn it into measurable criteria.
• Agentic and RAG evals
• Add evaluation of RAG, agents and multi-step tool use: tool-call correctness, task completion, groundedness and citation accuracy.
• Evaluate agent behaviour end to end through traces, not just final answers.
• Coding tool effectiveness
• Evaluate AI coding tools (e. g. Claude Code, Codex, Git Hub Copilot) on the firm’s own codebases and tasks.
• Measure how effective these tools are in practice: task success, code quality, review effort and delivery speed, and where they help or hinder.
• Turn results into practical guidance on tool choice, configuration and ways of working across engineering.
• Ground truth and datasets
• Build and maintain ground-truth datasets with PMs, quants and researchers.
• Keep datasets versioned, governed and reusable across teams.
• Evals in the delivery pipeline
• Run experiment tracking and integrate evals into CI/CD, so model, prompt and retrieval changes are gated on measured quality.
• Make the framework easy for other engineers to adopt by default.
• Assurance and governance
• Support red-teaming, failure analysis and the evidence needed for governance approval before deployment.
• Monitor quality in production and catch regressions early.
• Wider AI engineering
• Pick up other priorities across the team as they arise, from new front office tools to knowledge services, MCP integrations and data access.
• Contribute to shared engineering standards, code review and production support.
• Business outcomes
• Measure performance, accuracy and cost together, giving clear evidence for model, provider and adoption decisions.
• What Success Looks Like
• First 6 months
• Evaluation framework extended to cover the firm’s core AI assistants.
• First coding assistant evaluation on firm codebases, with recommendations adopted.
• Eval guidance and training in place for engineers building AI tools.
• First 12 months
• Evals gating releases in CI/CD for the firm’s AI assistants and agents.
• Eval tooling used by engineers across the firm by default.
• Coding tool effectiveness measured routinely, informing tooling and ways of working.
• Key Competencies

You are a senior engineer with a scientist’s instinct. You want to know whether something actually works, and you design tests that would show you if it didn’t.

You understand how LLMs fail, and you are rigorous about what a result does and does not prove.

You turn complex results into clear recommendations for engineers, governance forums and senior stakeholders, and you are comfortable saying when a tool or model is not good enough.

You enjoy teaching others why evals matter, and are happy to switch to whatever matters most to the team when priorities change.

Essential

• Key Experience and Qualifications
• 5+ years in software engineering, with strong Python and SQL.
• Hands‑on experience evaluating LLM systems: LLM-as-judge, human‑labelled datasets, retrieval metrics and agent trace analysis.
• Hands‑on experience with eval and observability tooling (e. g. Azure AI Foundry evals, Langfuse, Lang Smith, MLflow, promptfoo, Inspect).
• Strong testing discipline and an experimental mindset: baselines, controls, significance and honest reporting of results.
• Experience building LLM applications: retrieval (RAG), tool use and agents.
• Heavy daily use of agentic coding tools, with a clear view of where they fail and how to measure them.
• A track record of shipping production systems: APIs, containers, CI/CD and cloud (Azure or Kubernetes preferred).
• The ability to explain evaluation concepts and results clearly, and to bring teams with you.

Desirable

• Measuring developer productivity or the impact of engineering tooling.
• Benchmarking coding agents on real‑world software tasks.
• Red‑team ing and AI safety testing.
• Front office, market data or research experience at a hedge fund or bank.

Blue Crest is committed to providing an inclusive environment for its workforce.

As an employer, we provide equal opportunities to all people regardless of their gender, marital or civil partnership status, race, religion or ethnicity, disability, age, sexual orientation or nationality.