At a Glance
- Tasks: Design and implement automated QA strategies for GenAI features using Python.
- Company: Leading AI-infused planning platform serving Fortune 50 customers.
- Benefits: Competitive salary, flexible work options, and opportunities for professional growth.
- Other info: Join a dynamic team and shape the future of AI testing.
- Why this job: Make a real impact on cutting-edge AI technology and workflows.
- Qualifications: Experience in QA or test engineering with a focus on AI/ML systems.
The predicted salary is between 49500 - 60500 £ per year.
Overview
In this role you will design and implement automated QA strategies for Gen AI features, ensuring reliable, accurate, and trustworthy AI-powered workflows.
You will work across evaluations, APIs, UIs, and performance tests to validate non-deterministic AI systems.
You’ll shape evaluation frameworks for Gen AI metrics like accuracy and safety, and help scale testing within a cross-functional AI-focused team.
This is a hands-on, impact-driven position at a leading AI-infused planning platform that serves Fortune 50 customers.
You will translate complex AI behavior into robust test coverage and actionable insights.
Responsibilities
- Develop and execute end-to-end automated test coverage across evaluations, APIs, UIs, and performance tests using Python
- Implement testing strategies for Gen AI features, including conversational AI and agentic systems
- Maintain automated test suites for prompt testing and regression tests to detect model behavior changes
- Support evaluation frameworks measuring Gen AI performance across accuracy, relevance, safety, consistency, and latency
- Curate high-quality test datasets and golden examples representing diverse user scenarios
- Assist in adversarial testing to identify hallucinations or biases and support production alerting integration
- Utilise internal tools and testing frameworks to help developers validate Gen AI code and assist with UAT
- Document testing procedures and known issues for automated coverage workstream
- Key requirements
- Solid QA or test engineering experience with exposure to AI/ML systems
- Hands-on Python coding experience
- Knowledge of Gen AI technologies including LLMs, prompt engineering, and AI application patterns
- Experience with test automation frameworks and scripting (Python, Pytest, API, UI Testing)
- Understanding of functional, integration, regression, performance, and security testing
- Ability to execute test cases for non-deterministic systems
- Experience with API testing tools (Postman, REST Assured) and backend testing
- Familiarity with CI/CD pipelines and automated testing integration
- Python
- Pytest
- API testing
- UI testing
- Postman
- REST Assured
SDET - AI in London employer: Anaplan
Anaplan is an exceptional employer that fosters a Winning Culture, where innovation and collaboration thrive. Located in London, we offer a dynamic work environment that champions diversity and provides ample opportunities for professional growth, ensuring that every employee feels valued and inspired to contribute to our mission of transforming business decision-making. With a commitment to inclusivity and a focus on celebrating achievements, Anaplan is the perfect place for those looking to make a meaningful impact while advancing their careers.
