Full-stack Software Engineer (Product) in London

We are building Watcher, a coding agent security product. Watcher is deployed in production and monitors billions of agent tokens per month across engineering teams at agent-building scale-ups and enterprises. We are looking for a product engineer to build and scale Watcher end to end. This role includes backend and frontend components. You will own features across the full stack, from agent hooks and ingestion pipelines to monitoring dashboards and enterprise integrations. You can expect to own features from customer conversation to production deployment. This is truly a start-up role: you will own big chunks of the product, make decisions fast, and ship at high volume. You will join a small team with significant ability to shape the product and tech, and you can earn more responsibility quickly. This is an individual contributor role but could lead to management responsibilities eventually, if desired.

KEY RESPONSIBILITIES

• Ship product end to end (~45%)
• Feature development
• Design, build, and ship Watcher features across the stack: agent-side hooks and CLI tooling, ingestion and grading pipelines, real-time monitoring UI (Watcher Live), and the organization-wide Analyzer dashboard.
• Own features end to end, from ambiguous requirements to production: talk to users, scope the work, make the technical decisions, put that into production, and iterate based on usage.
• Build Watcher integrations: We currently support Claude Code and Codex, but we’re planning on increasing our coverage to other coding agents, LLM gateways, and anywhere that could benefit from agent monitoring.
• Build the policy and configuration layer that lets security teams set org-wide guardrails for engineers to operate safely within.
• Productionize research
• Turn monitoring research (new monitors, grading strategies, backtesting results) into production-ready product features. We are “our own customer”: You will use the product you build every day.
• Infrastructure and scale (~35%)
• Design and operate backend systems that process large volumes of agent logs in real time.
• Own reliability: robust error handling, graceful degradation, and observability (logging, metrics, tracing, alerting). We should catch issues before customers do.
• Architect data models and storage that handle both high-throughput writes and complex analytical queries over historical trajectories. Retention policies have to fit highly sensitive customer data.
• Make sure Watcher is safe to deploy: Work with our AI Security & Control Engineer on tenant isolation, encryption, and access controls. Watcher touches some of the most sensitive data our customers have.
• Customer and integration engineering (~20%)
• Build and maintain integrations with the enterprise security stack, e.g. streaming monitoring alerts into SIEM systems and existing security operations workflows.
• Support flexible deployment models: cloud-hosted, on-prem, and local backends, so sensitive agent logs never have to leave customer control.
• Talk to customers, debug issues in their environments, and feed what you learn back into the roadmap.

REPRESENTATIVE PROJECTS

• Secure a new frontier coding agent: Take an agent we don’t yet support (e.g. Cursor) from zero to fully monitored. Build the hook/integration layer, normalize its trajectory format into our data model, validate monitor accuracy on real traffic, and ship it to customers.
• Build the real-time blocking path: Design the low-latency pipeline that lets monitors block dangerous actions (e.g. git push -force, secret exfiltration) before they execute, while keeping p95 latency low enough that developers don’t feel it. Handle traffic spikes and partial failures gracefully.
• Ship an Analyzer capability end to end: Design and build an organization-wide view (e.g. failure trends over time, cross-session pattern detection) from data model through API to UI, based on what security teams actually need.
• Deploy Watcher inside locked-down enterprises: Build and document the self-hosted/on-prem deployment story so that enterprises with strict data requirements can run Watcher inside their own infrastructure.

JOB REQUIREMENTS

Must-haves

• 4+ years building production software.
• Full-stack capability.
• High agency and ownership.
• High output volume.
• Startup pace.
• Product sense.

Strong nice-to-haves

• Previous work on developer tools, monitoring/observability systems, or security products.
• Real-time or large-scale data processing experience.
• Enterprise or on-prem deployment experience.
• Experience building LLM-powered applications.
• Early-stage startup experience.

Explicitly not required

• Formal AI safety background.
• Management experience.
• Deep experience in every technology we use.

BENEFITS

• This role offers market competitive salary, equity, and competitive benefits.
• Flexible work hours and schedule.
• Unlimited vacation.
• Unlimited sick leave.
• Up to 6 months of paid parental leave.
• Comprehensive health, dental and vision insurance.
• Retirement savings with competitive employer matching.
• Lunch, dinner, and snacks are provided for all employees on workdays.
• Paid work trips, including staff retreats, business trips, and relevant conferences.
• A yearly $1,000 (USD) professional development budget.
• Relocation support and visa fees (if applicable).

LOGISTICS

• Time Allocation: Full-time.
• Location: This is an in-person role working out of our London or San Francisco office. We offer flexible working hours and some wfh arrangements.
• Visa sponsorship: We sponsor visas in both the UK and US.

ABOUT THE TEAM

The product team consists of product engineers and research scientists. Furthermore, you will interact with our other SWEs and researchers, since we intend to be our own customer by using our products internally for our research work.

ABOUT APOLLO RESEARCH

The rapid rise in AI capabilities offers tremendous opportunities, but also presents significant risks. At Apollo Research, we’re primarily concerned with risks from Loss of Control. We work closely with many frontier AI companies to test their models and collaborate on the science of scheming. At Apollo, we aim for a culture that emphasizes truth-seeking, being goal-oriented, giving and receiving constructive feedback, and being friendly and helpful.

Equality Statement: Apollo Research is an Equal Opportunity Employer. We value diversity and are committed to providing equal opportunities to all.