At a Glance
- Tasks: Design and implement automated testing frameworks for cutting-edge AI systems.
- Company: Join a forward-thinking tech company in the heart of London.
- Benefits: Competitive salary, hybrid work model, and potential for contract extension.
- Other info: Collaborative environment with opportunities for professional growth.
- Why this job: Be at the forefront of AI technology and ensure its reliability and performance.
- Qualifications: Strong Python skills and experience with test automation frameworks.
The predicted salary is between 60000 - 80000 £ per year.
This is a Fixed Term Contract until the end of 2026 but is highly likely to extend. We are looking for an AI Test Automation Engineer to join on a fixed-term contract until the end of the year, with potential to extend. The role is based in London, with some office presence required. You will help design and implement automated testing and evaluation frameworks for MCP Servers, LLM-based systems, and agentic AI workflows, ensuring accuracy, safety, reliability, and performance across complex AI-enabled platforms.
Key Responsibilities
- Develop automated testing frameworks for MCP Servers and related AI systems.
- Design evaluation strategies for LLM accuracy, safety, and reliability.
- Build automated tests using Python, Pytest, and BDD frameworks.
- Identify and address AI failure modes including hallucination, latency, and incorrect tool usage.
- Work with engineering, QA, and product teams to define quality metrics and acceptance criteria.
- Support Agile delivery, ensuring testing aligns with sprint goals.
- Produce reporting on quality outcomes, risks, and improvement areas.
- Maintain documentation for test cases, evaluation pipelines, and validation strategies.
Requirements
- Strong Python programming experience for test automation and evaluation.
- Expertise in Pytest and familiarity with BDD tools such as Behave or Cucumber.
- Knowledge of LLM evaluation approaches such as RAGAS, DeepEval, or custom evaluation pipelines.
- Understanding of agentic AI issues including hallucination, tool misuse, and performance bottlenecks.
- Experience testing AI workflows, distributed systems, or microservices environments.
- Strong knowledge of Agile delivery and CI/CD quality integration.
- Excellent communication, analytical, and problem-solving skills.
Nice to Have
- Experience with Model Context Protocol (MCP) or agent orchestration solutions.
- Exposure to observability, monitoring, or logging tools for AI systems.
- Background in API and service integration testing.
- Knowledge of containerised and cloud-native environments.
- Experience in enterprise AI automation or intelligent platform engineering.
Test Automation Specialist employer: Robson Bale
As a leading innovator in AI technology, we offer our employees a dynamic and collaborative work environment in the heart of London. Our commitment to professional development ensures that you will have ample opportunities for growth and advancement, while our hybrid work model promotes a healthy work-life balance. Join us to be part of a forward-thinking team that values creativity, diversity, and the pursuit of excellence in AI solutions.
StudySmarter Expert Advice🤫
We think this is how you could land Test Automation Specialist
✨Tip Number 1
Network like a pro! Reach out to folks in the industry, attend meetups, and connect with people on LinkedIn. You never know who might have the inside scoop on job openings or can refer you directly.
✨Tip Number 2
Show off your skills! Create a portfolio showcasing your test automation projects, especially those using Python and Pytest. This will give potential employers a taste of what you can do and set you apart from the crowd.
✨Tip Number 3
Prepare for interviews by brushing up on common questions related to AI testing and automation. Practice explaining your experience with LLM evaluation and Agile delivery, so you can confidently demonstrate your expertise.
✨Tip Number 4
Don’t forget to apply through our website! We’ve got some fantastic opportunities waiting for you, and applying directly can sometimes give you a better chance at landing that dream role.
We think you need these skills to ace Test Automation Specialist
Some tips for your application 🫡
Tailor Your CV:Make sure your CV is tailored to the Test Automation Specialist role. Highlight your Python programming skills and experience with Pytest, as these are key for us. Don’t forget to mention any relevant projects or frameworks you've worked on!
Craft a Compelling Cover Letter:Your cover letter is your chance to shine! Use it to explain why you're passionate about AI test automation and how your skills align with our needs. Be specific about your experience with LLM evaluation and Agile delivery.
Showcase Your Problem-Solving Skills:In your application, give examples of how you've tackled challenges in previous roles. We love seeing how you’ve identified and addressed issues like AI failure modes or performance bottlenecks in your past work.
Apply Through Our Website:We encourage you to apply directly through our website. It’s the best way for us to receive your application and ensures you’re considered for the role. Plus, it shows you’re keen on joining our team!
How to prepare for a job interview at Robson Bale
✨Know Your Tech Inside Out
Make sure you brush up on your Python skills and get familiar with Pytest and BDD frameworks like Behave or Cucumber. Be ready to discuss how you've used these tools in past projects, as well as any specific challenges you've faced and how you overcame them.
✨Understand AI Evaluation Metrics
Since the role involves evaluating LLM accuracy and safety, it’s crucial to understand various evaluation approaches like RAGAS and DeepEval. Prepare to explain these concepts and how they apply to ensuring the reliability of AI systems during the interview.
✨Showcase Your Agile Experience
This position requires a solid understanding of Agile delivery. Be prepared to share examples of how you've contributed to sprint goals and collaborated with engineering and QA teams to define quality metrics. Highlight any experience you have with CI/CD integration as well.
✨Communicate Clearly and Confidently
Strong communication skills are essential for this role. Practice articulating your thoughts clearly, especially when discussing complex topics like AI workflows and testing strategies. Remember, it's not just about what you know, but how you convey that knowledge!
