Senior QA & Automation Engineering Leader – AI Assurance and Test Automation in Glasgow

About Resillon

Resillion is a global provider of digital assurance, cybersecurity, and testing services. We help organisations strengthen their security posture, protect critical assets, and deliver high-quality digital solutions. Our teams operate across multiple specialist disciplines, including Digital Forensics, Incident Response, SOC Operations, Quality & Compliance, and Technical Consultancy. We are committed to excellence, continuous improvement, and delivering trusted, customer-focused services that meet the highest professional and accreditation standards.

Brief Description

About Resillon

Resillion is a global provider of digital assurance, cybersecurity, and testing services. We help organisations strengthen their security posture, protect critical assets, and deliver high-quality digital solutions. Our teams operate across multiple specialist disciplines, including Digital Forensics, Incident Response, SOC Operations, Quality & Compliance, and Technical Consultancy. We are committed to excellence, continuous improvement, and delivering trusted, customer-focused services that meet the highest professional and accreditation standards.

Senior QA & Automation Engineering Leader – AI Assurance and Test Automation

Central Scotland with Hybrid Travel to Glasgow

• PLEASE NOTE – DUE TO BACKGROUND CHECKS, WE CAN ONLY ACCEPT APPLICANTS WHO HAVE AT LEAST 3 YEARS UK ADDRESS HISTORY

Job Description

Role Overview

Senior QA and Automation Engineering Leader with over 20 years of experience defining and delivering quality engineering strategies across enterprise-scale programmes in pensions, banking, insurance, retail, and digital transformation.

Specialises in test automation architecture, data assurance, AI model quality assurance, and functional assurance of AI-enabled systems. Experienced in developing risk-based testing strategies, validating data quality and model outputs, and assessing the functional correctness, reliability, robustness, and responsible behaviour of Generative AI, Large Language Models (LLMs), and agentic AI systems.

Combines hands‑on technical expertise with strategic QA leadership to establish effective assurance frameworks, integrate continuous testing into CI/CD pipelines, and support the reliable and responsible deployment of complex enterprise applications and AI solutions.

Key Responsibilities

Data Assurance in the Context of AI Systems

• Define and implement data assurance strategies to establish the quality, integrity, completeness, consistency, accuracy, and representativeness of data used by AI systems.
• Perform data profiling and validation to identify missing values, anomalies, duplicates, outliers, and potential data quality issues across training, validation, and production datasets.
• Assess datasets for potential bias, imbalance, and underrepresentation, working with Data Engineering and Data Science teams to improve data suitability for AI model development and evaluation.
• Validate data pipelines, transformations, integrations, and ETL processes to ensure data is fit for purpose throughout the AI lifecycle.
• Establish data quality metrics, validation rules, reconciliation processes, and traceability between source data, transformed datasets, and AI outputs.
• Assess data governance, privacy, security, and access controls to support compliant and responsible AI implementation.
• Monitor data quality and identify data drift or changes in production data that may affect AI model performance and reliability.

AI Model Quality Assurance

• Define and execute AI model quality assurance strategies covering accuracy, relevance, consistency, robustness, reliability, and suitability for intended business use.
• Collaborate with Data Science and Machine Learning teams to validate model requirements, evaluation criteria, acceptance thresholds, and expected outcomes.
• Design and execute test scenarios to assess model behaviour across normal, boundary, negative, edge-case, and adversarial conditions.
• Evaluate AI-generated outputs for correctness, completeness, relevance, consistency, hallucinations, and alignment with business requirements.
• Assess model performance across representative datasets and user scenarios, including fairness, bias, explainability, and potential unintended outcomes.
• Establish repeatable evaluation approaches, benchmark datasets, regression suites, and measurable quality criteria for AI model releases.
• Assess changes in model behaviour over time and support monitoring for model degradation, data drift, and performance deviations.
• Document findings, defects, risks, and limitations, providing evidence-based recommendations to support model release and production readiness decisions.

Functional Assurance Requirements Of AI Systems

• Define functional assurance requirements for AI-enabled applications, ensuring that systems meet business objectives, user expectations, and specified acceptance criteria.
• Translate business requirements, AI use cases, and system specifications into testable functional requirements, assurance criteria, and end-to-end test scenarios.
• Validate AI system responses against expected behaviour, including accuracy of responses, contextual relevance, consistency, completeness, and adherence to defined business rules.
• Assess functional behaviour across LLM-based applications, Retrieval-Augmented Generation (RAG) solutions, Microsoft Copilot, and agentic AI workflows.
• Validate prompt handling, context management, retrieval accuracy, response generation, tool invocation, workflow orchestration, and integration with downstream enterprise systems.
• Test AI system behaviour under ambiguous inputs, conflicting instructions, incomplete information, unexpected user interactions, and changing contextual conditions.
• Assess failure handling, fallback mechanisms, human-in-the-loop controls, escalation procedures, and safe responses when confidence is low or outputs are unreliable.
• Validate guardrails and controls against prompt injection, unsafe tool usage, unauthorised actions, and unintended agent behaviour.
• Establish requirements traceability between business objectives, AI functional requirements, test scenarios, test evidence, and identified defects.
• Support release readiness assessments by evaluating functional risks, outstanding defects, acceptance criteria, and the adequacy of assurance evidence.

Test Automation Architecture and Continuous Testing

• Architect and maintain scalable automation frameworks using Selenium WebDriver, Playwright, Cypress, UFT, Cucumber BDD, Python, and Java.
• Develop automated UI, API, mobile, backend, batch, and data validation test suites.
• Integrate automated testing into Azure DevOps, Jenkins, and GitHub Actions CI/CD pipelines, implementing quality gates and automated reporting.
• Apply AI‑assisted testing approaches using GitHub Copilot and Microsoft Copilot to accelerate test design, code generation, and test maintenance, with appropriate human review and validation.
• Improve automation coverage, regression efficiency, defect detection, and release confidence through risk‑based test execution.

API, Integration, Data Migration, and Non‑Functional Testing

• Design and execute API and integration tests covering authentication, OAuth 2.0, token‑based access, session management, and service interactions.
• Lead backend data validation, database reconciliation, migration testing, and ETL assurance across Oracle, DB2, MySQL, and enterprise data platforms.
• Validate end-to-end business workflows and data exchanges across interconnected enterprise systems.
• Support performance, resilience, and security readiness testing using tools such as JMeter and established OWASP principles.
• Develop risk‑based mobile testing strategies for iOS and Android applications, including React Native and hybrid application architectures, using Appium and Python.

Quality Governance and Stakeholder Leadership

• Define QA governance standards, risk‑based assurance strategies, quality metrics, and evidence‑based release acceptance criteria.
• Work closely with business stakeholders, solution architects, software engineers, Data Engineering, and Data Science teams to manage quality risks.
• Establish defect management, requirements traceability, test reporting, and assurance documentation practices.
• Mentor distributed QA and automation engineering teams, promoting technical excellence and continuous improvement.
• Communicate AI and non‑AI system risks, quality findings, limitations, and recommendations to technical and business stakeholders.

Technical Skills

AI and Data Assurance: Data quality profiling, data integrity, representativeness, bias assessment, model output validation, hallucination evaluation, data and model drift, AI functional assurance, RAG and agentic AI testing concepts.

Test Automation: Selenium WebDriver, Playwright, Cypress, UFT/QTP, Appium, Cucumber BDD.

Programming: Python, PyTest, Java, JavaScript, TypeScript, C#, VBScript.

API and Integration: Postman, RestAssured, SoapUI, OAuth 2.0

CI/CD and DevOps: Azure DevOps, Jenkins, GitHub Actions, Git, Maven, Gradle, Docker.

Data Platforms: SQL, Oracle, DB2, MySQL, ETL validation, data profiling, Databricks.

Performance and Security: JMeter, LoadRunner, OWASP principles.

Governance and Delivery: Risk-based testing, requirements traceability, Agile, Scrum, SAFe, Jira, Xray, Confluence, ServiceNow.

#J-18808-Ljbffr