Senior Research Engineer, LLM Training & Post-Training in London

Lightning AI is the company behind PyTorch Lightning. Founded in 2019, we build an end-to-end platform for developing, training, and deploying AI systems—designed to take ideas from research to production with less friction. We serve solo researchers, startups, and large enterprises. Lightning AI operates globally with offices in New York City, San Francisco, Seattle, and London.

The people who thrive here are builders who move fast, communicate openly, take ownership, and continuously improve themselves, their teams, and our company. Here's what that looks like in practice:

• Move with Urgency: We move quickly, make thoughtful decisions, and keep momentum.
• Take Ownership: We own outcomes, not just our individual work.
• Communicate Openly: We communicate directly, seek to understand, and create clarity for others.
• Build Great Teams: We lead by example, empower others, and create healthy teams.
• Raise the Bar: We're always improving ourselves.
• Think Long-Term: We design for what's next.

We're looking for an experienced Senior Research Engineer who has built, trained, and optimized modern transformer-based language models to join our Research Engineering function at Lightning. This role will focus on advancing how large language models are trained, fine-tuned, evaluated, and deployed across Lightning AI's platform and real-world customer workloads. It will work across model training, post-training, PyTorch, distributed systems, and AI systems engineering to improve model quality, training efficiency, and developer productivity while collaborating closely with researchers, infrastructure engineers, and customers.

This role is hybrid with a minimum of 2 in-office days per week in San Francisco, Seattle, NYC, or London, with fully remote work considered for candidates outside of our office hub locations.

What You'll Do:

• Design, build, and optimize training and post-training pipelines for large language models.
• Improve model quality through supervised fine-tuning, continued pretraining, preference optimization, reinforcement learning, evaluation, and experimentation.
• Build and improve PyTorch-based training infrastructure, tooling, and developer workflows.
• Optimize distributed training across multi-GPU environments.
• Investigate model training issues, including convergence, instability, communication overhead, and performance bottlenecks.
• Design evaluation methodologies, benchmark models, analyze failure modes, and achieve model improvements through experimentation.
• Collaborate directly with customers to understand real-world workloads.
• Partner closely with research, infrastructure, and platform engineering teams to build production-ready AI systems.
• Contribute to open-source projects through new features, tooling improvements, documentation, and community engagement.

What You’ll Need:

• Significant experience training, fine-tuning, evaluating, and/or optimizing transformer-based language models using PyTorch.
• Experience with modern LLM training and post-training techniques.
• Strong understanding of distributed training and multi-node systems.
• Strong software engineering fundamentals.
• Experience designing experiments, evaluating model performance, and debugging complex training and/or optimization issues.
• Excellent communication and collaboration skills.
• Comfortable working in fast-moving, ambiguous environments.
• Master's degree, PhD, or equivalent industry experience in Machine Learning, AI, Computer Science, or a related field.

Ideal Experience:

• Experience with one or more of the following: DeepSpeed, FSDP, Megatron-LM, Hugging Face Transformers, TRL, PEFT, Lightning Fabric, NVIDIA Molt, or similar training frameworks.
• CUDA, Triton, vLLM, SGLang, TensorRT, or other AI systems and performance optimization technologies.
• Open-source contributions, research publications, or production AI platforms supporting large-scale training or inference workloads.
• Startup experience or experience working on highly cross-functional engineering teams.

Compensation: We are committed to offering competitive compensation that reflects the value each team member brings to our mission.

Benefits and Perks: We offer a comprehensive and competitive benefits package designed to support our employees’ health, well-being, and long-term success:

• Comprehensive Health Coverage: Medical, dental, and vision coverage for employees and eligible dependents.
• Meaningful Equity: RSUs that give employees a stake in the company's long-term success.
• Retirement Savings: 401(k) matching (U.S.) and pension contributions (U.K.).
• Flexible Time Off: Unlimited PTO, company holidays, and floating holidays.
• Company-Wide Winter Break: Two weeks of company closure each winter.
• Paid Parental & Family Leave: Paid leave to support you and your family.
• Professional Development: Annual learning and development allowance.
• Wellness Benefits: Wellness and work-from-home stipends.
• Sabbatical Program: Four weeks of paid sabbatical leave after four years of service.
• Flexible Work: Flexible schedules and a hybrid work model.
• In-Office Meals: Complimentary meals at our office hubs.

At Lightning AI, we are committed to fostering an inclusive and diverse workplace. We believe that diverse teams drive innovation and create better products.