ML Systems Engineer - Transferable Intelligence Team in City of London

• ML Systems Engineer
• Transferable Intelligence Team
• The Company

Octai Pipe is a young, ambitious company with the vision to be the global driving force for the next paradigm of foundational, physical AI that ensures our connected world, and its critical infrastructure, is safe, secure and sustainable.

We are growing fast, having closed a recent funding round and looking to accelerate rapidly.

Octai Pipe is offering the right candidate an exciting role on this adventure!

Octai Pipe is on a mission to revolutionise the optimisation of energy in data centres through decentralised artificial intelligence (AI).

To do this, Octai Pipe is harnessing an elegant but revolutionary idea.

Rather than move the data from the source, move the algorithms to the data to learn at the data source.

This learning can be achieved with the intelligence of many devices through novel federated AI technology.

Octai Pipe is developing the AI for Cooling Efficiency (ACE) application to be deployed using its own in-house distributed AI platform.

The Role

The Transferable Intelligence team builds the capability that lets what Octai Pipe's AI learns at one site benefit others: reinforcement learning agents for control, and causal inference and root cause analysis models, that train at the edge and transfer between heterogeneous data centre environments.

We are looking for an ML Systems Engineer to own the systems engineering behind this.

Models are deployed through a central portal that orchestrates what runs on each edge device.

The devices are heterogeneous on-prem hardware inside data centres, running a demanding mix of concurrent workloads: RL policy inference, fine-tuning, an LLM-based explainer, and on-device analytics served through a UI.

This demands disciplined systems engineering: the right models, at the right versions, running efficiently within their resource budgets, across the entire fleet.

The role also has a secondary research dimension.

You will work side-by-side with the team's applied and research scientists as a hands-on force multiplier: building the training systems and experiment infrastructure that take new algorithms and system designs from idea to result quickly, reproducibly and at scale, much of it within our staging environment of large virtual device fleets.

Duties and Responsibilities

• Own workload coordination on the device: scheduling, prioritisation, resource budgets and isolation across RL inference, fine-tuning, the LLM explainer and analytics.
• Guarantee model lifecycle correctness across the fleet: versioning, atomic model swaps, and reconciliation between portal intent and device state, working closely with the internal tools team building our model registry.
• Build the team's deployment workflows on the central portal: packaging, staged rollout, shadow-mode deployment, guardrails, and automatic fallback and rollback.
• Help build and scale the staging environment and the training systems inside it: continuous, distributed and federated training and fine-tuning across real and staging fleets, with automated orchestration, checkpointing, recovery and cross-site adaptation workflows.
• Profile and optimise on-device performance across heterogeneous hardware, including efficient LLM inference.
• Develop monitoring, alerting and observability on our Grafana and Elastic stack, enabling early detection of drift, anomalies and degradation.
• Build the telemetry and data pipelines that feed on-device analytics, causal inference and root cause analysis, with the data quality and lineage those methods demand.
• Debug failures across the full stack and unblock applied and research scientists quickly.
• Raise the engineering bar through code review, documentation, runbooks and post-mortems.

Your Profile

• Strong software engineering fundamentals and excellent Python.
• Experience building and operating end-to-end ML systems in production, from data pipelines through training to serving and monitoring.
• Solid distributed systems experience with care for correctness: state consistency, versioning and reconciliation across unreliable networks and heterogeneous hardware.
• Experience coordinating multiple workloads on shared hardware: resource budgeting, scheduling and isolation.
• Experience optimising model inference for production (profiling, quantisation, compilation) to run efficiently on constrained or shared hardware.
• Deep working knowledge of the ML training stack (Py Torch or similar), including distributed training and checkpoint/recovery.
• Experience building monitoring and observability with Grafana and the Elastic Stack, or similar.
• Skilled at debugging complex failures across the stack.
• Pragmatic, low-ego and hands-on; comfortable with autonomy while collaborating closely with scientists and engineers.
• You Also Might Have
• Reinforcement learning infrastructure experience (e. g. Ray/RLlib, environment APIs, simulators) or deployment of closed-loop control systems.
• Experience building simulation or virtual test environments for distributed systems.
• Familiarity with LLM serving stacks (e. g. v LLM, llama. cpp) or optimisation toolchains such as ONNX Runtime and Tensor RT.
• Experience with federated learning, transfer learning or privacy-preserving ML.
• Experience with on-prem GPU infrastructure or network-restricted environments.
• Experience with edge fleet management: device provisioning, remote deployment and over-the-air updates.
• Experience with telemetry and streaming systems (Kafka, MQTT).
• Exposure to security and compliance in critical infrastructure or regulated environments.
• Why Join Octai Pipe
• Work on real-world sustainability impact at global scale
• Influence how AI is responsibly applied to critical infrastructure
• Join a well-funded, rapidly growing scale-up with ambitious goals
• Collaborate with experts across AI, infrastructure, and operations
• Shape a product that can materially reduce energy use and carbon emissions worldwide

The above statements are not intended to encompass all functions and qualifications of the position; rather, they are intended to provide a general framework of the requirements of the position.

Job incumbents may be required to perform other functions not specifically addressed in this description.