At a Glance
- Tasks: Test and validate cutting-edge Generative AI products to ensure quality and reliability.
- Company: Join a multi-award winning tech company with a vibrant culture.
- Benefits: Enjoy 25 days holiday, private medical insurance, and a supportive work environment.
- Other info: Great progression opportunities and a commitment to diversity and inclusion.
- Why this job: Be at the forefront of AI innovation and make a real impact in technology.
- Qualifications: 3+ years in QA or software testing, with experience in AI/ML applications.
The predicted salary is between 45000 - 55000 £ per year.
We’re looking for an AI Quality Analyst to help validate and improve Generative AI products and features. This role focuses on testing LLM-based systems, prompt workflows, agent behaviours, RAG pipelines, and AI-driven user experiences. You will work closely with product, engineering, and AI teams to identify quality risks, design robust test strategies, and ensure our GenAI solutions are reliable, safe, and production‑ready.
VARIED DAY‑TO‑DAY RESPONSIBILITIES
- Design and execute test plans for Generative AI applications, including chat, summarization, extraction, classification, and agent workflows.
- Validate LLM outputs for quality, relevance, accuracy, consistency, tone, and policy compliance.
- Test prompt behaviour across normal, edge, and adversarial scenarios.
- Evaluate RAG systems for retrieval quality, grounding, hallucination risk, and citation accuracy.
- Create test cases for multi‑step AI workflows, tool use, memory, and orchestration logic.
- Partner with engineers and product managers to define acceptance criteria for AI features.
- Identify and document defects, regressions, safety issues, and model failure patterns.
- Build and maintain test datasets, evaluation rubrics, and benchmark suites.
- Support automated testing for AI features, APIs, and end‑to‑end workflows.
- Track model and prompt changes to assess quality impact over time.
- Contribute to release readiness by reporting on AI quality metrics, risks, and recommendations.
WHAT ARE WE LOOKING FOR IN A CANDIDATE?
- 3+ years in QA, software testing, or test engineering.
- Hands‑on experience testing AI/ML or Generative AI applications.
- Strong understanding of LLM behaviour, including hallucinations, prompt sensitivity, grounding, and non‑deterministic outputs.
- Experience writing detailed test cases, bug reports, and test summaries.
- Familiarity with API testing tools and test automation frameworks.
- Ability to evaluate model outputs using structured rubrics and human judgment.
- Strong analytical skills and attention to detail.
- Comfortable working cross‑functionally in a fast‑moving product environment.
Preferred Qualifications
- Experience testing chatbots, copilots, agents, or RAG‑based applications.
- Familiarity with prompt engineering and prompt/version evaluation.
- Experience with Python, SQL, or scripting for test data and automation.
- Knowledge of AI safety, red teaming, bias testing, and policy evaluation.
- Experience with model evaluation frameworks, telemetry, or experimentation.
- Exposure to cloud AI platforms and LLM providers.
WHAT IS LIFE LIKE AT SMARTSEARCH?
- We are a multi‑award winning tech company with an aspirational mentality.
- Recognitions include being named on the RegTech100 list for 2024, listed in the Top 100 Fast‑Growing Tech Companies by Northern Tech Awards 2024, and named Technology Provider of the Year by Corporate Finance Awards 2024.
- Great Place To Work certified since 2022.
- Excellent progression opportunities with personal development goals, regular feedback, and support.
- Diverse and inclusive team committed to promoting diversity & inclusion and social responsibility, with active DE&I initiatives and community support.
COMPANY BENEFITS
- 25 days holiday rising to 30 with each year of service.
- Private medical insurance covering dental and optical.
- Company pension scheme.
- Life assurance – 4× your annual salary.
- 1 day paid volunteering per year.
- Enhanced maternity / paternity offerings.
- Employee assistance programme.
- Cycle to work scheme.
- Access to a gym.
AI Tester employer: Occupop
At SmartSearch, we pride ourselves on being a multi-award winning tech company that fosters an aspirational and inclusive work culture. As an AI Tester, you will benefit from excellent progression opportunities, a supportive environment focused on personal development, and a range of perks including private medical insurance and generous holiday allowances. Join us in West Yorkshire, where your contributions to cutting-edge Generative AI products will be valued and recognised, making this an ideal place for meaningful and rewarding employment.
StudySmarter Expert Advice🤫
We think this is how you could land AI Tester
✨Tip Number 1
Network like a pro! Reach out to folks in the AI and tech space on LinkedIn or at local meetups. We all know that sometimes it’s not just what you know, but who you know that can help you land that dream job.
✨Tip Number 2
Prepare for those interviews by brushing up on your knowledge of LLMs and Generative AI. We suggest doing mock interviews with friends or using online platforms to get comfortable discussing your experience and skills.
✨Tip Number 3
Showcase your skills through a portfolio! If you’ve worked on any AI testing projects, compile them into a neat presentation. We love seeing real examples of your work, so don’t hold back!
✨Tip Number 4
Apply directly through our website! It’s the best way to ensure your application gets seen by the right people. Plus, we’re always on the lookout for passionate candidates who are eager to join our team.
We think you need these skills to ace AI Tester
Some tips for your application 🫡
Tailor Your Application:Make sure to customise your CV and cover letter for the AI Tester role. Highlight your experience with QA, software testing, and any hands-on work with AI/ML applications. We want to see how your skills align with what we're looking for!
Showcase Your Testing Skills:When writing your application, be specific about your testing experience. Mention any detailed test cases, bug reports, or test summaries you've created. This will help us understand your analytical skills and attention to detail.
Be Clear and Concise:Keep your application clear and to the point. Use bullet points where possible to make it easy for us to read. We appreciate a well-structured application that gets straight to the important stuff!
Apply Through Our Website:Don't forget to apply through our website! It’s the best way for us to receive your application and ensures you’re considered for the role. Plus, it shows you're keen on joining our team at StudySmarter!
How to prepare for a job interview at Occupop
✨Know Your AI Stuff
Make sure you brush up on your knowledge of Generative AI and LLM behaviour. Understand concepts like hallucinations, prompt sensitivity, and grounding. Being able to discuss these topics confidently will show that you're not just familiar with the field but also passionate about it.
✨Prepare Test Scenarios
Think about various test scenarios you might encounter in the role. Prepare examples of how you would design and execute test plans for different AI applications. This will demonstrate your analytical skills and attention to detail, which are crucial for an AI Quality Analyst.
✨Showcase Your Collaboration Skills
Since this role involves working closely with product, engineering, and AI teams, be ready to discuss your experience in cross-functional collaboration. Share specific examples of how you've partnered with others to define acceptance criteria or resolve quality issues in past projects.
✨Ask Insightful Questions
Prepare some thoughtful questions about the company's AI products and their testing processes. This shows your genuine interest in the role and helps you understand how you can contribute to their goals. Plus, it gives you a chance to assess if the company is the right fit for you!
