Job Responsibilities
- Define and implement comprehensive quality assurance strategies and test plans for AI agents and LLM-powered applications
- Design and develop robust, scalable, and maintainable automated test frameworks
- Collaborate with product managers, machine learning engineers, and data scientists to translate AI features and model behaviors into test cases and validation criteria
- Improve testing processes and infrastructure, integrating automated checks into CI/CD pipelines
- Identify, document, track, and analyze software defects and inconsistencies
- Monitor production systems and AI model performance and contribute to post-release quality validation
- Champion quality best practices across engineering teams
- Design, manage, and maintain test data strategies and mock services
- Manage the full defect lifecycle using customer feedback and debugging logs
- Report directly to the director of engineering
Requirements
- 5+ years of hands-on experience in software quality assurance or engineering, with a strong focus on testing complex distributed systems or AI/ML applications
- 5+ years of experience working with GitHub Actions
- Proficiency in Python or TypeScript for test automation
- Experience with modern testing frameworks such as Playwright
- Solid understanding of AI/ML concepts, including model evaluation metrics, data pipelines, and the unique challenges of testing generative AI outputs
- Experience designing, developing, or integrating agentic AI systems, AI skills, and the Model Context Protocol (MCP)
- Experience with AWS, Azure, or GCP and Docker or Kubernetes in a CI/CD environment is a big plus
- Exceptional analytical skills and keen attention to detail
- Demonstrated ability to drive initiatives independently and thrive in a fast-paced, evolving startup environment
- Must be located within 50 miles of one of the office hubs or be willing to relocate
- Must be able and willing to work in person in the office 3 days per week
- Must be legally authorized to work in the country in which the job is located
Core Competencies
Demonstrates expertise in quality assurance strategies for AI and LLM applications, with a strong focus on automated testing frameworks and CI/CD integration. Proficient in Python or TypeScript, with a solid understanding of AI/ML concepts and testing methodologies.
Highest-signal resume keywords
- Quality Assurance Strategies
- Automated Test Frameworks
- Python or TypeScript
- AI/ML Concepts
- GitHub Actions
Hard Skills
- Quality Assurance
- Test Automation
- Automated Testing Frameworks
- AI/ML Testing
- Defect Lifecycle Management
- Model Evaluation Metrics
- Data Pipelines
- Agentic AI Systems
- Playwright
- CI/CD Integration
Soft Skills
- Analytical Skills
- Attention to Detail
- Initiative
- Adaptability
- Collaboration
Industry Keywords
- AI Agents
- LLM-Powered Applications
- Testing Complex Distributed Systems
- Mock Services
- Post-Release Quality Validation
Tools & Technologies
- GitHub Actions
- AWS
- Azure
- GCP
- Docker
- Kubernetes
#J-18808-Ljbffr
Software Quality Engineer employer: Jobtailor
As a Client Services Coordinator at our dynamic company, you will thrive in a supportive work culture that prioritises employee growth and development. We offer comprehensive training, opportunities for advancement, and a collaborative environment where your contributions are valued. Located in a vibrant area, our team enjoys a healthy work-life balance and the chance to engage with diverse clients, making every day rewarding and meaningful.
