Senior Machine Learning Engineer (IntelliJ AI)

We are currently looking to hire a Senior Machine Learning Engineer to help us realize our ambitious vision of creating AI assistance that supports the entire development lifecycle across JetBrains IDEs. If selected, you will join the ML subteam within IntelliJ AI, driving the development of our ML system from end to end by defining evaluation and metrics, shaping context orchestration, and helping product teams tailor AI capabilities to their needs.

In This Role, You Will:

• Design and drive evaluation frameworks for AI features, including metrics, experiments, and agent trace analysis.
• Diagnose model performance issues (e.g. prompt drift, context mismatches, and latency/quality trade-offs) and translate findings into actionable improvements.
• Experiment with contexts and lightweight models to continuously develop our ML system.
• Act as the ML liaison for product teams across JetBrains, adapting and scaling AI capabilities in JetBrains IDEs to their needs.
• Build and maintain small helper models (e.g. re-rankers, classifiers, embedding models) to support domain‑specific tasks.
• Collaborate with colleagues in ML, product, engineering, and analytic teams to deliver improvements and monitor their impact in production.
• Stay up to date with research in the fields of LLMs, agents, and evaluation, bringing best practices into our workflows.
• Mentor junior engineers and help shape team culture, processes, and tooling around experimentation and evaluation.

Qualifications:

• Have 5+ years of experience as an ML Engineer, with a solid background in production‑grade ML systems (especially LLMs and agent architectures).
• Have experience with LLM evaluation methods and frameworks.
• Can design and run end‑to‑end experiments – hypotheses, metrics, data collection (including traces/logs), analysis, and decision‑making.
• Are skilled in context‑aware pipelines or conversational/agent systems.
• Have strong Python programming skills.
• Bring hands‑on experience in fine‑tuning or training smaller models (e.g. domain‑specific fine‑tuning and lightweight customizations).
• Communicate clearly and effectively across teams, translating ML/AI insights into product features.
• Have prior mentorship experience with ML/evaluation engineers.
• Thrive in a cross‑functional, fast‑moving environment, taking ownership, iterating quickly, and delivering results.

Preferred Qualifications:

• Familiarity with agent‑based systems and orchestrating multi‑step reasoning agents.
• Experience with the Kotlin programming language.

Location: London, England, United Kingdom