At a Glance
- Tasks: Build and scale a cutting-edge coding agent security product from end to end.
- Company: Join an innovative tech company focused on AI safety and security.
- Benefits: Enjoy competitive salary, unlimited vacation, flexible hours, and professional development budget.
- Other info: Dynamic team culture with opportunities for rapid career growth and learning.
- Why this job: Make a real impact in a fast-paced startup environment with ownership of significant features.
- Qualifications: 4+ years in software development with full-stack capabilities and a passion for innovation.
The predicted salary is between 63000 - 77000 £ per year.
We are building Watcher, a coding agent security product. Watcher is deployed in production and monitors billions of agent tokens per month across engineering teams at agent-building scale-ups and enterprises. We are looking for a product engineer to build and scale Watcher end to end. This role includes backend and frontend components. You will own features across the full stack, from agent hooks and ingestion pipelines to monitoring dashboards and enterprise integrations. You can expect to own features from customer conversation to production deployment. This is truly a “start-up role”: you will own big chunks of the product, make decisions fast, and ship at high volume. You will join a small team with significant ability to shape the product and tech, and you can earn more responsibility quickly. This is an individual contributor role but could lead to management responsibilities eventually, if desired.
KEY RESPONSIBILITIES
- Ship product end to end (~45%)
- Design, build, and ship Watcher features across the stack: agent-side hooks and CLI tooling, ingestion and grading pipelines, real-time monitoring UI (Watcher Live), and the organization-wide Analyzer dashboard.
- Own features end to end, from ambiguous requirements to production: talk to users, scope the work, make the technical decisions, put that into production, and iterate based on usage.
- Build Watcher integrations: We currently support Claude Code and Codex, but we’re planning on increasing our coverage to other coding agents, LLM gateways, and anywhere that could benefit from agent monitoring.
- Build the policy and configuration layer that lets security teams set org-wide guardrails for engineers to operate safely within.
- Productionize research: Turn monitoring research (new monitors, grading strategies, backtesting results) into production-ready product features.
- We are "our own customer": You will use the product you build every day.
- Infrastructure and scale (~35%)
- Design and operate backend systems that process large volumes of agent logs in real time.
- Own reliability: robust error handling, graceful degradation, and observability (logging, metrics, tracing, alerting). We should catch issues before customers do.
- Architect data models and storage that handle both high-throughput writes and complex analytical queries over historical trajectories. Retention policies have to fit highly sensitive customer data.
- Make sure Watcher is safe to deploy: Work with our AI Security & Control Engineer on tenant isolation, encryption, and access controls. Watcher touches some of the most sensitive data our customers have.
- Customer and integration engineering (~20%)
- Build and maintain integrations with the enterprise security stack, e.g. streaming monitoring alerts into SIEM systems and existing security operations workflows.
- Support flexible deployment models: cloud-hosted, on-prem, and local backends, so sensitive agent logs never have to leave customer control.
- Talk to customers, debug issues in their environments, and feed what you learn back into the roadmap.
REPRESENTATIVE PROJECTS
- Secure a new frontier coding agent: Take an agent we don't yet support (e.g. Cursor) from zero to fully monitored. Build the hook/integration layer, normalize its trajectory format into our data model, validate monitor accuracy on real traffic, and ship it to customers.
- Build the real-time blocking path: Design the low-latency pipeline that lets monitors block dangerous actions (e.g. git push --force, secret exfiltration) before they execute, while keeping p95 latency low enough that developers don't feel it. Handle traffic spikes and partial failures gracefully.
- Ship an Analyzer capability end to end: Design and build an organization-wide view (e.g. failure trends over time, cross-session pattern detection) from data model through API to UI, based on what security teams actually need.
- Deploy Watcher inside locked-down enterprises: Build and document the self-hosted/on-prem deployment story so that enterprises with strict data requirements can run Watcher inside their own infrastructure.
JOB REQUIREMENTS
Must-haves
- 4+ years building production software. You have shipped and operated real systems with real users, and you know what it takes to keep them reliable.
- Full-stack capability. You are strong on the backend (APIs, data pipelines, databases, cloud infrastructure) and productive on the frontend (modern web UI). You don't need to be world-class at both, but you must be able to own a feature across the whole stack. Our stack is primarily Python and TypeScript.
- High agency and ownership. You are willing to own big chunks of the product, make decisions fast with incomplete information, and be accountable for the outcome. You don't wait to be told what to do and have an accurate sense of the roadmap for the product.
- High output volume. You have used Claude Code, Codex, Cursor, or similar tools heavily to accelerate or have built agents or agent tooling yourself.
- Startup pace. You are excited about a fast-moving environment, comfortable with ambiguity and changing priorities, and willing to grind when it matters.
- Product sense. You can talk to users, translate vague requirements into concrete designs, and make good calls about what to build and what to cut.
Strong nice-to-haves
- Previous work on developer tools, monitoring/observability systems, or security products. Watcher sits at the intersection of all three.
- Real-time or large-scale data processing experience (streaming pipelines, message queues, high-throughput log systems).
- Enterprise or on-prem deployment experience. Shipping software into environments you don't control is its own skill.
- Experience building LLM-powered applications, e.g. LLM-as-judge setups, evaluation pipelines, or familiarity with frameworks like Inspect.
- Early-stage startup experience. You have been employee 1-15 somewhere and know what it feels like to build a product with no playbook.
Explicitly not required
- Formal AI safety background. We need excellent product engineers who can learn the AI safety context, not AI safety researchers who need to learn engineering.
- Management experience. This is an IC role, at least initially.
- Deep experience in every technology we use. We care about demonstrated ability to learn and ship, not checkbox familiarity with our exact stack.
BENEFITS
This role offers market competitive salary, equity, and competitive benefits.
- Salary: San Francisco: $222,000 – $290,000 London: £149,000 – £195,000
- Our engineers effectively have an unlimited token budget. If a better result costs more compute, use it.
- Flexible work hours and schedule
- Unlimited vacation
- Unlimited sick leave
- Up to 6 months of paid parental leave
- Comprehensive health, dental and vision insurance
- Retirement savings with competitive employer matching (e.g. 401(k) for US employees)
- Lunch, dinner, and snacks are provided for all employees on workdays
- Paid work trips, including staff retreats, business trips, and relevant conferences
- A yearly $1,000 (USD) professional development budget
- Relocation support and visa fees (if applicable)
LOGISTICS
- Time Allocation: Full-time
- Location: This is an in-person role working out of our London or San Francisco office. We offer flexible working hours and some wfh arrangements.
- Visa sponsorship: We sponsor visas in both the UK and US. Sponsorship isn't guaranteed for every role or candidate, but if we make you an offer, we'll work with you to find the right visa route.
ABOUT THE TEAM
The product team consists of product engineers: Jeremy Neiman, Zak Walters, Zen van Riel, Srdjan Miletic and Gustavo Bicalho; research scientists: Victor Gillioz, Monika Jotautaitė, Dmitrii Volkov; and our GTM lead: Kyle Dai. Marius Hobbhahn (CEO) advises the team. Furthermore you will interact with our other SWEs and researchers, since we intend to be "our own customer" by using our products internally for our research work.
ABOUT APOLLO RESEARCH
The rapid rise in AI capabilities offers tremendous opportunities, but also presents significant risks. At Apollo Research, we're primarily concerned with risks from Loss of Control, i.e. risks coming from the model itself rather than e.g. humans misusing the AI. We're particularly concerned with deceptive alignment / scheming, a phenomenon where a model appears to be aligned but is, in fact, misaligned and capable of evading human oversight. We work on the science of scheming, detection of scheming (e.g. building evaluations), and scheming mitigations (e.g. anti-scheming). We also work on control and monitoring research (see our scalable monitoring agenda). We work closely with many frontier AI companies, such as OpenAI, Anthropic, Google, Meta, Thinking Machines and others, e.g. to test their models and collaborate on the science of scheming. At Apollo, we aim for a culture that emphasizes truth-seeking, being goal-oriented, giving and receiving constructive feedback, and being friendly and helpful.
If you're interested in more details about what it's like working at Apollo, you can find more information here.
We also build a coding agent security product called Watcher that secures agent deployments in companies. Our goal is to reduce the probability of catastrophic incidents by securing coding agents, learning about their real-world risks, and publishing our research on how to build these control systems most effectively.
Equality Statement: Apollo Research is an Equal Opportunity Employer. We value diversity and are committed to providing equal opportunities to all, regardless of age, disability, gender reassignment, marriage and civil partnership, pregnancy and maternity, race, religion or belief, sex, or sexual orientation.
HOW TO APPLY
Please complete the application form with your CV. The provision of a cover letter is neither required nor encouraged. Please also feel free to share links to relevant work samples.
About the interview process: Our multi-stage process includes a screening interview, a take-home test (3 hours), 3 technical interviews, and a final interview with Marius (CEO). There are no leetcode-style general coding interviews. You may use AI tools on the take-home; we judge the result the way we'd judge any contributor's work, so you are responsible for the quality of everything you submit. If you want to prepare, we suggest building simple monitors for coding agents and running them on your own Claude Code / Cursor / Codex / etc. traffic.
Your Privacy and Fairness in Our Recruitment Process: We are committed to protecting your data, ensuring fairness, and adhering to workplace fairness principles in our recruitment process. To enhance hiring efficiency, we use AI-powered tools to assist with tasks such as resume screening. These tools are designed and deployed in compliance with internationally recognized AI governance frameworks. Your personal data is handled securely and transparently. All resumes are screened by a human and final hiring decisions are made by our team. If you have questions about how your data is processed or wish to report concerns about fairness, please contact us at [email protected].
Full-stack Software Engineer (Product) employer: Apollo Research
At Apollo Research, we pride ourselves on fostering a dynamic and innovative work environment where engineers can take ownership of significant product features and make impactful decisions. Our London office offers competitive salaries, unlimited vacation, and a strong emphasis on professional development, ensuring that our team members grow alongside our cutting-edge projects in AI security. With a culture that values truth-seeking and collaboration, you'll be part of a small, agile team dedicated to shaping the future of coding agent security.
StudySmarter Expert Advice🤫
We think this is how you could land Full-stack Software Engineer (Product)
✨Join Local Tech Meetups
Get out there and mingle with fellow developers by joining local tech meetups. It’s a fantastic way to meet people who might be working at Apollo Research or know someone who does. Plus, you can pick up some trendy tech skills and trends while you're at it!
✨Contribute to Open Source Projects
Show off your coding chops by jumping into open-source projects. Not only does this give you practical experience, but it also gets you noticed in the dev community. You'll create a killer portfolio that speaks volumes about your skills to Apollo Research.
✨Tap into Online Developer Communities
Don’t underestimate the power of online developer communities like GitHub, Stack Overflow, and even Reddit. Participate in discussions, share your projects, and build your visibility. We can often find opportunities through these channels that can lead to a full-time gig at companies like Apollo Research.
✨Explore Job Boards Specifically for Tech Roles
Keep your eyes peeled on job boards that focus on tech roles. Sites like TechCareers or Stack Overflow Jobs can often have listings for companies like Apollo Research that might not show up on broader job sites. Make it a habit to check these regularly, and don’t hesitate to apply directly through our website!
We think you need these skills to ace Full-stack Software Engineer (Product)
Some tips for your application 🫡
Show off your coding skills:When applying for a software engineering role, it's super important to showcase your coding skills. Make sure your CV includes your tech stack, any relevant programming languages you’re comfortable with, and examples of projects you've worked on. If you have a GitHub profile, link it up! We love to see code in action.
Tailor your portfolio:For a full-time role, we’d expect to see some solid examples of your work in your portfolio. Make sure to include at least two or three projects that highlight your problem-solving skills and your ability to work with different technologies. Focus on the projects that are most relevant to the position at Apollo Research.
Craft a killer cover letter:Your cover letter is your chance to stand out—make it personal! Explain why you want to work at Apollo Research and how your skills align with the role. Show us your passion for software development. We dig enthusiastic candidates who understand the value of collaboration and continuous learning!
Be clear and concise:When it comes to writing your CV and cover letter, clarity is key. Avoid jargon that could confuse us and stick to simple, direct language. Highlight your achievements with quantifiable results where possible, and keep everything easy to read. A well-organised application goes a long way!
How to prepare for a job interview at Apollo Research
✨Brush Up on Your Coding Skills
For a full-time software engineering role, it's crucial that we stay sharp with our coding abilities. Expect technical questions that might involve solving problems on the spot or discussing algorithms. Practise on platforms like LeetCode or HackerRank to get comfortable with the types of questions that often come up.
✨Know Your Tools and Frameworks
Make sure we’re well-acquainted with the tools and technologies listed in the job description. Familiarise ourselves with any specific frameworks or programming languages mentioned. If Apollo Research uses React or Node.js, for instance, be ready to discuss how we’ve used them in previous projects or coursework.
✨Showcase Your Projects
Bring along a portfolio that highlights our best work. This could be code samples, GitHub repositories, or any side projects we’ve built. Make sure we can talk through our thought process for each project, especially the challenges we faced and how we solved them—this shows our problem-solving skills in action.
✨Prepare for Behavioural Questions
While technical skills are key, full-time positions also require cultural fit. Be ready to discuss our previous experiences and how we handle teamwork, conflict, and deadlines. Brush up on the STAR method—Situation, Task, Action, Result—to clearly articulate our past experiences when discussing how we've contributed to a team.