At a Glance
- Tasks: Design coding tasks for AI agents and verify their solutions in a Docker environment.
- Company: Mindrift connects specialists with exciting project-based AI opportunities.
- Benefits: Earn up to $35/hour, flexible hours, and work on innovative projects.
- Other info: Enjoy a dynamic work environment with opportunities for top performers.
- Why this job: Challenge yourself by creating real-world coding scenarios for cutting-edge AI technology.
- Qualifications: 3+ years in backend software development and Python fluency required.
The predicted salary is between 35 - 35 £ per hour.
Description
Please submit your CV in English and indicate your level of English proficiency.
Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems.
Participation is project-based, not permanent employment.
About the Role
You’ll design coding tasks that challenge frontier AI coding agents.
Each task is a self-contained Docker environment with a broken piece of software; an AI agent attempts the fix; automated tests verify the outcome.
Your deliverable is the full task package: broken code, tests, instructions, and a reference solution proving the task is solvable.
Responsibilities
- Invent a realistic developer scenario — a real bug, a broken ETL, a missing feature — not a toy problem.
- Build a reproducible Docker environment with pinned dependencies.
- Write a pytest that verifies outcomes, not specific commands — deterministic, non-flaky, and does not leak the fix.
- Write an instruction. md that reads like a Jira ticket a developer would receive.
- Write a reference solve. sh proving the task is solvable.
- Calibrate difficulty so current state-of-the-art agents solve the task 20–60% of the time.
- Iterate based on feedback from expert QA reviewers.
- Later: review other authors’ tasks as a QA reviewer.
- Not in scope
- Data labeling, prompt engineering.
- Production code to ship — you design problems and verification for AI agents.
- Leetcode puzzles — scenarios must look like real developer work.
- Not every candidate task ships — quality over quantity.
Requirements
3+ years of production software development in one backend stack — Python, Go, Node. js, Java, or Rust. Depth in one stack beats breadth.
Python + pytest fluency — required regardless of primary stack.
The task harness is pytest-based even when the broken app is in another language. Fixtures, parametrize, monkeypatch, timeouts, conftest. py.
Docker authoring
— reproducible Dockerfiles, pinned dependencies, multi-stage builds when needed, non-root user.
Linux & Bash
— comfort debugging inside containers (strace, lsof, journalctl); shell beyond set -euo pipefail.
AI coding agent experience
— Claude Code, Cursor, Roo Code, or similar, on non-trivial work. You can cite a specific time the AI was confidently wrong and how you caught it.
English — B2+ written.
- Not a fit
- Data Science, ML, or Computer Vision engineers without backend-engineering output.
- Manual QA testers without automation or test authoring.
- Frontend-only, low-code / no-code, IT Support, or Business Analysts.
- Engineers who have never written pytest from scratch.
- Junior, intern, or assistant as the most recent role.
- Preferred qualifications
- Domain depth in Security, System Administration (nginx / systemd / cron), Scientific Computing (Num Py / Py Torch / Sci Py), Dev Ops, or Git internals.
- Modern Python tooling (uv, poetry, pyproject. toml).
- Coverage tooling (pytest-cov, coverage. py, gcov, llvm-cov, kcov).
- Fuzzing or property-based testing (Hypothesis).
- Prior contribution to agent-evaluation benchmarks or related frameworks.
- Process
Apply → Pass qualification (90-minute sample-task screen + short behavioral interview) → Join a project → Complete tasks → Get paid.
- Time commitment
- Onboarding: ~10 hours per first task.
- Steady state: ~5 hours per task, 2–4 parallel tasks per author.
- Realistic weekly load: 8–20 hours. Higher volume available for top performers.
- You choose when and how to contribute; tasks must be submitted by the deadline and meet acceptance criteria.
Compensation
- Paid contributions, rates up to
- Task-based compensation equivalent to hourly rate, depending on performance and volume.
- Some projects include incentive payments.
*Rates vary based on expertise, skills assessment, location, project needs, and other factors.
Higher rates may be provided to highly specialized experts.
Lower rates may apply during onboarding or non-core project phases.
Payment details are shared per project.
Apply
Submit your CV via the Mindrift platform.
Indicate your English level, note this role (Software Engineering Evaluation Specialist — Terminal Bench), and include a Git Hub profile link if available.
Software Engineering Evaluation Specialist employer: Mindrift
Mindrift is an excellent employer for those seeking a flexible and rewarding role in the innovative field of AI mortgage underwriting. With competitive compensation and a focus on part-time project-based work, employees benefit from a supportive work culture that values expertise and encourages professional growth. Located in Edinburgh, Mindrift offers a unique opportunity to contribute to cutting-edge technology while maintaining a healthy work-life balance.
StudySmarter Expert Advice🤫
We think this is how you could land Software Engineering Evaluation Specialist
✨Join Local Tech Meetups
Get out there and mingle with fellow developers by joining local tech meetups. It’s a fantastic way to meet people who might be working at Mindrift or know someone who does. Plus, you can pick up some trendy tech skills and trends while you're at it!
✨Contribute to Open Source Projects
Show off your coding chops by jumping into open-source projects. Not only does this give you practical experience, but it also gets you noticed in the dev community. You'll create a killer portfolio that speaks volumes about your skills to Mindrift.
✨Tap into Online Developer Communities
Don’t underestimate the power of online developer communities like GitHub, Stack Overflow, and even Reddit. Participate in discussions, share your projects, and build your visibility. We can often find opportunities through these channels that can lead to a full-time gig at companies like Mindrift.
✨Explore Job Boards Specifically for Tech Roles
Keep your eyes peeled on job boards that focus on tech roles. Sites like TechCareers or Stack Overflow Jobs can often have listings for companies like Mindrift that might not show up on broader job sites. Make it a habit to check these regularly, and don’t hesitate to apply directly through our website!
We think you need these skills to ace Software Engineering Evaluation Specialist
Some tips for your application 🫡
Show off your coding skills:When applying for a software engineering role, it's super important to showcase your coding skills. Make sure your CV includes your tech stack, any relevant programming languages you’re comfortable with, and examples of projects you've worked on. If you have a GitHub profile, link it up! We love to see code in action.
Tailor your portfolio:For a full-time role, we’d expect to see some solid examples of your work in your portfolio. Make sure to include at least two or three projects that highlight your problem-solving skills and your ability to work with different technologies. Focus on the projects that are most relevant to the position at Mindrift.
Craft a killer cover letter:Your cover letter is your chance to stand out—make it personal! Explain why you want to work at Mindrift and how your skills align with the role. Show us your passion for software development. We dig enthusiastic candidates who understand the value of collaboration and continuous learning!
Be clear and concise:When it comes to writing your CV and cover letter, clarity is key. Avoid jargon that could confuse us and stick to simple, direct language. Highlight your achievements with quantifiable results where possible, and keep everything easy to read. A well-organised application goes a long way!
How to prepare for a job interview at Mindrift
✨Brush Up on Your Coding Skills
For a full-time software engineering role, it's crucial that we stay sharp with our coding abilities. Expect technical questions that might involve solving problems on the spot or discussing algorithms. Practise on platforms like LeetCode or HackerRank to get comfortable with the types of questions that often come up.
✨Know Your Tools and Frameworks
Make sure we’re well-acquainted with the tools and technologies listed in the job description. Familiarise ourselves with any specific frameworks or programming languages mentioned. If Mindrift uses React or Node.js, for instance, be ready to discuss how we’ve used them in previous projects or coursework.
✨Showcase Your Projects
Bring along a portfolio that highlights our best work. This could be code samples, GitHub repositories, or any side projects we’ve built. Make sure we can talk through our thought process for each project, especially the challenges we faced and how we solved them—this shows our problem-solving skills in action.
✨Prepare for Behavioural Questions
While technical skills are key, full-time positions also require cultural fit. Be ready to discuss our previous experiences and how we handle teamwork, conflict, and deadlines. Brush up on the STAR method—Situation, Task, Action, Result—to clearly articulate our past experiences when discussing how we've contributed to a team.
