At a Glance
- Tasks: Define evaluation frameworks and develop automated test pipelines for AI services.
- Company: Global organisation leading in AI platform engineering.
- Benefits: Competitive daily rate, hybrid work model, and exposure to cutting-edge AI technology.
- Other info: Collaborative team environment with opportunities for professional growth.
- Why this job: Join a transformative AI initiative and shape the future of enterprise AI.
- Qualifications: Strong Python skills and experience with automated testing in regulated environments.
A global organisation is building a centralised AI enablement and platform engineering function focused on delivering scalable, secure, and governed AI capabilities across the enterprise. This role sits within a programme delivering enterprise-grade agentic AI infrastructure, including internal AI assistants, retrieval and search services, extensibility frameworks, and governance tooling.
The programme is focused on delivering a production-grade internal agentic AI platform, including:
- Development of an enterprise AI assistant capable of reasoning, planning, and tool orchestration
- Operation of enterprise retrieval, search, and grounding services for approved data sources
- Delivery of a secure internal gateway layer providing discovery, observability, policy enforcement, and lifecycle management for AI-integrated services
- Design and development of AI-integrated services and reusable capabilities that safely expose internal and third-party systems to AI agents
- Establishment of evaluation, governance, and quality-control frameworks to support scalable and compliant deployment of AI capabilities
The programme currently follows a centrally delivered model while evolving towards a federated contribution approach over time.
Key Responsibilities
- Define and implement evaluation frameworks covering correctness, safety, reliability, and regression impact for AI-integrated services
- Develop and maintain automated test pipelines for agentic workflows, including tool orchestration and multi-step execution paths
- Identify, evaluate, and mitigate AI system failure modes such as hallucinations, invalid inputs, latency issues, and inappropriate tool usage
- Produce testing and governance evidence required for internal approval and operational processes
- Collaborate closely with ML Engineers and platform teams to embed testability and evaluation capabilities into AI services
- Contribute to the long-term quality assurance and governance strategy for enterprise-wide AI platform adoption
Essential Skills and Experience
- Strong Python development experience, particularly for automation and test frameworks
- Experience with LLM and RAG evaluation tooling, frameworks, or custom evaluation pipelines
- Expertise in automated testing across unit, integration, and regression testing environments
- Good understanding of agentic AI systems, associated risks, and operational failure modes
- Ability to assess technical solutions against governance, audit, and security requirements
- Experience working within regulated or highly governed engineering environments
What’s on Offer
- Opportunity to contribute to large-scale enterprise AI transformation initiatives
- Exposure to cutting-edge AI platform engineering and governance challenges
- Collaborative environment working alongside platform engineers, ML specialists, and architecture teams
- Influence over the development of long-term AI quality and governance standards
- Opportunity to shape scalable AI engineering practices within a complex enterprise environment
Quality Engineer - AI (Contract). Job in London LilyLifestyle Jobs employer: United Cerebral Palsy of Georgia
As a Quality Engineer in our Central London office, you will be part of a dynamic team driving the future of AI technology within a global organisation. We offer a collaborative work culture that values innovation and professional growth, providing you with the opportunity to influence enterprise-wide AI quality standards while working on cutting-edge projects. With a hybrid work model and a focus on employee development, we ensure that your contributions are recognised and rewarded in a meaningful way.
Contact Details:
United Cerebral Palsy of Georgia Recruitment Team
StudySmarter Expert Advice🤫
We think this is how you could land Quality Engineer - AI (Contract). Job in London LilyLifestyle Jobs
✨Tip Number 1
Network like a pro! Reach out to folks in the AI and engineering space on LinkedIn or at industry events. A friendly chat can open doors that a CV just can't.
✨Tip Number 2
Show off your skills! If you’ve got a portfolio or some projects that highlight your Python and testing expertise, make sure to share them during interviews. It’s all about proving you can walk the walk.
✨Tip Number 3
Prepare for those tricky questions! Brush up on your knowledge of agentic AI systems and be ready to discuss how you’d tackle potential failure modes. Confidence is key!
✨Tip Number 4
Don’t forget to apply through our website! We’re always on the lookout for talent like yours, and applying directly can give you a leg up in the process.
We think you need these skills to ace Quality Engineer - AI (Contract). Job in London LilyLifestyle Jobs
Some tips for your application 🫡
Tailor Your CV:Make sure your CV is tailored to the Quality Engineer role. Highlight your Python development experience and any work you've done with AI systems. We want to see how your skills match what we're looking for!
Showcase Your Experience:In your application, don’t just list your previous jobs. Share specific examples of projects where you’ve implemented evaluation frameworks or automated testing. This helps us understand your hands-on experience in regulated environments.
Be Clear and Concise:When writing your application, keep it clear and to the point. Use bullet points for key achievements and avoid jargon unless it's relevant. We appreciate straightforward communication that gets to the heart of your qualifications.
Apply Through Our Website:We encourage you to apply through our website for a smoother process. It helps us keep track of applications and ensures you get all the updates directly from us. Plus, it’s super easy!
How to prepare for a job interview at United Cerebral Palsy of Georgia
✨Know Your AI Stuff
Make sure you brush up on your knowledge of agentic AI systems and their associated risks. Be ready to discuss how you've tackled issues like hallucinations or latency in past projects. This will show that you understand the technical challenges and can contribute effectively.
✨Showcase Your Python Skills
Since strong Python development experience is essential, prepare to talk about specific projects where you've used Python for automation and testing frameworks. Bring examples of automated test pipelines you've developed, as this will demonstrate your hands-on expertise.
✨Understand Evaluation Frameworks
Familiarise yourself with evaluation frameworks for AI-integrated services. Be prepared to discuss how you would define and implement these frameworks, focusing on correctness, safety, and reliability. This shows you're not just a coder but also a strategic thinker.
✨Collaborative Mindset
This role involves working closely with ML Engineers and platform teams, so highlight your teamwork skills. Share examples of how you've collaborated in the past to embed testability into AI services. This will illustrate your ability to work in a cross-functional environment.
