At a Glance
- Tasks: Design and develop AI evaluation tools to ensure quality and safety of AI systems.
- Company: Join a leading tech firm focused on innovative AI solutions.
- Benefits: Competitive daily rate, flexible work location, and opportunities for professional growth.
- Other info: Collaborative environment with a focus on responsible AI practices.
- Why this job: Make a real impact in the AI field while working with cutting-edge technologies.
- Qualifications: Strong experience in AI/ML evaluation and hands-on Python development skills.
The predicted salary is between 63000 - 77000 £ per year.
4 Months 750 per day (Inside IR35)
Security Clearance: Active SC Clearance preferred / eligible and willing to undergo SC Clearance.
Alternative backgrounds may include SDET, AI Evaluation Engineer, AI Assurance Engineer, Prompt Engineer, Harness Engineer or AI Quality Engineer.
This is a hands‑on engineering role for someone with deep expertise in AI/ML evaluation, testing and quality assurance, rather than a cloud, infrastructure or platform engineer.
You will play a key role in establishing robust evaluation frameworks for production‑grade AI systems, helping assure the quality, safety and effectiveness of data‑driven and AI‑enabled products.
Responsibilities
- Design, develop and deploy AI evaluation and risk management tooling.
- Build and maintain evaluation frameworks for AI, ML and LLM‑based products.
- Create and develop code solutions, harnesses and testing capabilities from the ground up using Python.
- Define and execute evaluation strategies across model, agentic and application layers.
- Assess AI system quality, performance, reliability, safety and risk.
- Work closely with technical and non‑technical stakeholders across multiple departments and organisations.
- Provide technical leadership and guidance on AI evaluation best practice.
Skills Experience
- Strong commercial experience working with AI, Machine Learning and LLM‑based systems in production environments.
- Deep understanding of AI evaluation methodologies, frameworks and quality engineering principles.
- Hands‑on Python development experience with the ability to create and maintain your own codebase.
• Experience evaluating AI solutions across
- Model layer
- Agentic layer
- Application layer
• Knowledge of AI evaluation tooling and frameworks such as
- RAGAS
- Deep Eval (DP-Eval)
- Other AI evaluation and testing frameworks
- Experience developing automated testing, evaluation harnesses and assurance processes for AI systems.
- Proven leadership experience, including working across teams, departments and organisational boundaries.
- Strong stakeholder engagement and communication skills.
Desirable Experience
- Public sector or government programme experience.
- Understanding of responsible AI and AI assurance frameworks.
- Please Note
This opportunity is specifically seeking candidates with strong AI evaluation and engineering expertise.
Experience primarily focused on cloud, infrastructure, platform or Dev Ops engineering without significant AI evaluation experience are unlikely to be suitable.
#J-18808-Ljbffr
AI Evaluation Engineer employer: SmartSourcing Ltd
As a Brand Strategist with our public sector client in London, you will be part of a dynamic team dedicated to shaping and protecting a unique brand that serves the community. The company fosters a collaborative work culture that values innovation and creativity, offering flexible hybrid working arrangements that promote work-life balance. With a commitment to employee growth, you will have access to professional development opportunities that empower you to excel in your role and make a meaningful impact on government services.
StudySmarter Expert Advice🤫
We think this is how you could land AI Evaluation Engineer
✨Get Involved in Open-Source Projects
Diving into open-source projects is a brilliant way to showcase your skills and connect with other developers in the community. Not only will you beef up your GitHub profile but you might also catch the eye of someone at SmartSourcing Ltd who values hands-on experience over just theory.
✨Attend Local Tech Meetups
Tech meetups are gold mines for networking and discovering job opportunities, especially in the fast-paced world of software engineering. Check out local listings for events in your area and don’t shy away from introducing yourself. This could lead directly to a temporary position at SmartSourcing Ltd!
✨Showcase Your Work Online
With temporary roles, it’s all about standing out in a short space of time. Create a portfolio website where you highlight your projects and skills. Talk about your code, and provide links to your GitHub repositories. This will not only demonstrate your abilities but will also make it easier for recruiters at SmartSourcing Ltd to see what you bring to the table.
✨Leverage Temporary Job Boards
Don’t forget to check specialised job boards for temporary software development roles. Websites like We Work Remotely or Remote OK often list short-term gigs that can be a perfect fit. Apply directly through our website as well, making sure your application is sharp—temporary roles can move fast!
We think you need these skills to ace AI Evaluation Engineer
Some tips for your application 🫡
Show Off Your Tech Skills:Make sure your CV highlights your tech stack and any programming languages you’re proficient in. Include specifics about any frameworks or technologies you’ve worked with; they can make you stand out in the sea of applicants. It’s all about showing that you have the chops we need at SmartSourcing Ltd!
Portfolio 2.0:Since you’re applying for a temporary gig, it’s super important to showcase a portfolio that highlights your best projects. Include links to GitHub or any personal projects that demonstrate what you can do in a real-world environment. This gives us a taste of your style and your problem-solving approach!
Keep It Brief and Relevant:With a temporary position, we want to see your ability to hit the ground running. Be concise in your CV and cover letter; stick to experiences that directly relate to the role. Highlight any previous temporary roles or freelance gigs that show your adaptability and quick learning!
Tailor Your Cover Letter:Don’t just send a generic cover letter. Personalise it for AI Evaluation Engineer at SmartSourcing Ltd! Mention why this temporary role excites you and how you see yourself contributing in the short run. Show us what you've got and why you're the one for this quick turn-around!
How to prepare for a job interview at SmartSourcing Ltd
✨Nail the Technical Skills
For a software engineering role, you'll likely face technical questions or coding tasks during your interview. Brush up on the relevant programming languages and frameworks that SmartSourcing Ltd uses, and don’t forget to practice some coding challenges on platforms like LeetCode or HackerRank. Showing your coding prowess can really make you stand out!
✨Prepare for System Design Questions
Even for a temporary role, having a grasp of system design principles can be crucial. Be ready to discuss how you would architect a software solution, including discussing trade-offs, scalability, and performance considerations. Having examples from previous projects can really show off your analytical thinking.
✨Demonstrate Your Adaptability
Since this is a temporary role, you'll want to emphasise your ability to hit the ground running. Highlight experiences where you quickly adapted to new technologies or teams. Let’s make it clear to the interviewers at SmartSourcing Ltd that you can learn on the job and deliver results in a short timeframe!
✨Show Off Your Portfolio
Make sure to have a portfolio or GitHub ready showcasing your projects. Having tangible evidence of what you've done—be it personal projects, contributions to open-source, or previous work—can convey how capable you are. Tailor this for what might interest SmartSourcing Ltd, so it's relevant and sparks conversation during your interview.
