At a Glance
- Tasks: Design and operate cutting-edge ML infrastructure for advanced engineering simulations.
- Company: PhysicsX, a deep-tech innovator transforming hardware innovation with AI.
- Benefits: Equity options, generous leave, private medical insurance, and wellness subscriptions.
- Other info: Diverse and inclusive workplace with excellent career development opportunities.
- Why this job: Join a team pushing the boundaries of engineering with AI-driven solutions.
- Qualifications: 5+ years in ML infrastructure, strong Python skills, and collaboration experience.
The predicted salary is between 80000 - 100000 £ per year.
About Us
PhysicsX is a deep‑tech company with roots in numerical physics and Formula One, dedicated to accelerating hardware innovation at the speed of software. We are building an AI‑driven simulation software stack for engineering and manufacturing across advanced industries. By enabling high‑fidelity, multi‑physics simulation through AI inference across the entire engineering lifecycle, PhysicsX unlocks new levels of optimization and automation in design, manufacturing, and operations – empowering engineers to push the boundaries of possibility. Our customers include leading innovators in Aerospace & Defense, Materials, Energy, Semiconductors, and Automotive.
The Role
The Principal ML Infrastructure Engineer will extend and operate the infrastructure that powers our research model training, fine‑tuning, and serving pipelines. You will be embedded within our Research function, partnering directly with ML engineers and research scientists to ensure they can train Large Physics Models efficiently and reliably at scale.
Team Context
- Research Scientists who determine the model architectures and methods
- ML Engineers who implement and develop the models
- Simulation Data Engineers who are accountable for upstream data pipelines
You will have end‑to‑end responsibilities over the research infrastructure, with the autonomy to make architectural decisions and the responsibility to keep data flowing reliably. Horizontally, you will be part of an infrastructure engineering group responsible for infrastructure across the company.
What you will do
Training Infrastructure
- Design and operate distributed training infrastructure for neural operator architectures (Transolver, Point Cloud Transformer, etc.) on our large NVIDIA DGX B200 platform.
- Optimize training pipelines for throughput, fault tolerance, and cost efficiency, including checkpointing strategies, gradient accumulation, and multi‑node synchronization.
- Build and maintain experiment tracking and observability systems that give researchers clear visibility into training runs, hyperparameter sweeps, and model performance.
Data I/O and Performance
- Solve data loading bottlenecks for large‑scale mesh datasets.
- Optimize data pipelines for efficient I/O from cloud storage, including prefetching, caching, and format optimization.
- Work with heterogeneous data sources of varying formats and resolutions.
Model Serving and Deployment
- Build serving infrastructure for pre‑trained LPMs, supporting both zero‑shot inference and uncertainty quantification (Monte Carlo Dropout).
- Design and implement model packaging pipelines for customer deployment. Models must run reliably in customer environments with fine‑tuning capabilities.
- Ensure reproducibility: any model checkpoint should be deployable with consistent behaviour.
Platform and Tooling
- Improve developer experience for the Research team with fast iteration cycles, reliable CI/CD, clear debugging tools.
- Collaborate with the broader Infrastructure team on shared patterns and standards.
What you bring to the table
- Ability to scope and effectively deliver projects, prioritising activity as needed.
- Problem‑solving skills and the ability to analyse issues, identify causes, and recommend solutions quickly.
- Excellent collaboration and communication skills, especially in a research setting.
- You can translate "the model isn't converging" into infrastructure hypotheses and solutions, and can bridge technical abstractions with implementations.
- 5+ years of experience building and operating ML infrastructure at scale:
- Deep expertise in distributed training: debugged NCCL hangs, optimised collective communication, and know when to use FSDP vs. DDP vs. pipeline parallelism.
- Strong systems fundamentals: Linux, networking (including domain specific NVLink and InfiniBand), storage I/O, profiling and performance optimisation.
- Production experience with Kubernetes and SLURM for job orchestration on GPU clusters.
- Proficiency in Python and ML frameworks (PyTorch strongly preferred).
- Experience with cloud GPU infrastructure; ideally CoreWeave or similar GPU/HPC‑focused clouds.
- Ideally
- Experience with geometric deep learning or neural operators, architectures that operate on meshes, point clouds, or graphs.
- Background in HPC for simulation engineering, familiarity with how CFD/FEA workflows generate and consume data.
- Experience building model serving infrastructure with latency and throughput requirements.
- Familiarity with experiment tracking tools (Weights & Biases, MLflow) and observability stacks (Prometheus, Grafana).
- Experience packaging models for deployment into customer environments (containers, model registries, versioning).
What We Offer
- Equity options – share meaningfully in the company you’re helping to build.
- 10% employer pension contribution.
- Free office lunches.
- Enhanced parental leave – 3 months full pay paternity and 6 months full pay maternity leave.
- YellowNest nursery scheme – to help working parents manage childcare costs.
- 25 days of annual leave (+ public holidays).
- Private medical insurance – 100% employee cover.
- Wellhub subscription – access to thousands of gyms, classes and wellness apps.
- Eye tests.
- Personal development – dedicated support for learning, development and leveling up over time.
- Employee Assistance Programme (EAP) – confidential wellbeing support.
- Bike2Work scheme and season ticket loan.
- Octopus EV salary sacrifice – for a simpler, more sustainable way to drive electric.
We value diversity and are committed to equal employment opportunity regardless of sex, race, religion, ethnicity, nationality, disability, age, sexual orientation or gender identity. We strongly encourage individuals from groups traditionally underrepresented in tech to apply. To help make a change, we sponsor bright women from disadvantaged backgrounds through their university degrees in science and mathematics. We collect diversity and inclusion data solely for the purpose of monitoring the effectiveness of our equal opportunities policies and ensuring compliance with UK employment and equality legislation. This information is confidential, used only in aggregate form, and will not influence the outcome of your application.
Principal Machine Learning Infrastructure Engineer in London employer: PhysicsX
PhysicsX is an exceptional employer, offering a dynamic work environment in the heart of London where innovation meets opportunity. As a Finance Assistant, you'll gain invaluable hands-on experience in a high-growth deep-tech company, with a strong emphasis on employee development and a culture that values accuracy and proactivity. With hybrid working arrangements and a commitment to diversity and inclusion, PhysicsX empowers its employees to thrive while contributing to groundbreaking advancements in engineering and manufacturing.
StudySmarter Expert Advice🤫
We think this is how you could land Principal Machine Learning Infrastructure Engineer in London
✨Join Local Tech Meetups
Get out there and mingle with fellow developers by joining local tech meetups. It’s a fantastic way to meet people who might be working at PhysicsX or know someone who does. Plus, you can pick up some trendy tech skills and trends while you're at it!
✨Contribute to Open Source Projects
Show off your coding chops by jumping into open-source projects. Not only does this give you practical experience, but it also gets you noticed in the dev community. You'll create a killer portfolio that speaks volumes about your skills to PhysicsX.
✨Tap into Online Developer Communities
Don’t underestimate the power of online developer communities like GitHub, Stack Overflow, and even Reddit. Participate in discussions, share your projects, and build your visibility. We can often find opportunities through these channels that can lead to a full-time gig at companies like PhysicsX.
✨Explore Job Boards Specifically for Tech Roles
Keep your eyes peeled on job boards that focus on tech roles. Sites like TechCareers or Stack Overflow Jobs can often have listings for companies like PhysicsX that might not show up on broader job sites. Make it a habit to check these regularly, and don’t hesitate to apply directly through our website!
We think you need these skills to ace Principal Machine Learning Infrastructure Engineer in London
Some tips for your application 🫡
Show off your coding skills:When applying for a software engineering role, it's super important to showcase your coding skills. Make sure your CV includes your tech stack, any relevant programming languages you’re comfortable with, and examples of projects you've worked on. If you have a GitHub profile, link it up! We love to see code in action.
Tailor your portfolio:For a full-time role, we’d expect to see some solid examples of your work in your portfolio. Make sure to include at least two or three projects that highlight your problem-solving skills and your ability to work with different technologies. Focus on the projects that are most relevant to the position at PhysicsX.
Craft a killer cover letter:Your cover letter is your chance to stand out—make it personal! Explain why you want to work at PhysicsX and how your skills align with the role. Show us your passion for software development. We dig enthusiastic candidates who understand the value of collaboration and continuous learning!
Be clear and concise:When it comes to writing your CV and cover letter, clarity is key. Avoid jargon that could confuse us and stick to simple, direct language. Highlight your achievements with quantifiable results where possible, and keep everything easy to read. A well-organised application goes a long way!
How to prepare for a job interview at PhysicsX
✨Brush Up on Your Coding Skills
For a full-time software engineering role, it's crucial that we stay sharp with our coding abilities. Expect technical questions that might involve solving problems on the spot or discussing algorithms. Practise on platforms like LeetCode or HackerRank to get comfortable with the types of questions that often come up.
✨Know Your Tools and Frameworks
Make sure we’re well-acquainted with the tools and technologies listed in the job description. Familiarise ourselves with any specific frameworks or programming languages mentioned. If PhysicsX uses React or Node.js, for instance, be ready to discuss how we’ve used them in previous projects or coursework.
✨Showcase Your Projects
Bring along a portfolio that highlights our best work. This could be code samples, GitHub repositories, or any side projects we’ve built. Make sure we can talk through our thought process for each project, especially the challenges we faced and how we solved them—this shows our problem-solving skills in action.
✨Prepare for Behavioural Questions
While technical skills are key, full-time positions also require cultural fit. Be ready to discuss our previous experiences and how we handle teamwork, conflict, and deadlines. Brush up on the STAR method—Situation, Task, Action, Result—to clearly articulate our past experiences when discussing how we've contributed to a team.