Principal Data Engineer

About the role

The Principal Data Engineer will design and build a highly scalable, resilient data platform to support high-volume trading workloads. They will also provide self-service capabilities and influence technology strategy across the organization's data and platform engineering functions.

Requirements

Candidates must have over 10 years of software development experience, specifically with JVM-based languages and large-scale distributed systems. Expertise in modern data lakehouse architectures, real-time streaming, and cloud-native infrastructure is essential for this role.

Full description

Shift Pattern: Standard 40 Hour Week (United Kingdom)

Scheduled Weekly Hours: 40

Corporate Grade: C - Vice President

Reporting Line: (UK Division) Information Technology

Location: UK-London

Worker Type: Permanent

The purpose of this role is to help shape the LME's strategic data platform which can reliably process, store, and serve high-volume, high-velocity trading data at scale, while providing engineering teams with reusable, self-service capabilities that accelerate the delivery of trading and analytical solutions.

Success in this role means delivering a highly scalable and resilient data platform that becomes the engineering foundation for all data-intensive trading workloads, enabling faster innovation, improved reliability, reduced operational complexity, and better utilisation of market and trading data across the organisation.

Strong Required Demonstrable Experience of:

• A candidate for this position must have got working experience in IT organisation as a technologist who has evolved from a strong background in hands on software development.
• 10+ years of experience with software development, preferably working with JVM based languages (Java/Scala/Kotlin).
• Provide expertise on the development of highly scalable data platforms, data warehouses (SQL server, Postgres), and modern data lakes.
• Solve complex scalability, reliability, latency, and performance challenges associated with large-scale distributed systems.
• Provide self-service platform capabilities that allow engineering teams to onboard, process, discover, and consume data efficiently.
• Influence technology strategy and roadmaps across Trading Technology, Data Engineering, and Platform Engineering functions.
• Build high-volume distributed data platforms capable of processing billions of events per day and managing terabytes scale datasets.
• Develop modern Data Lakehouse platforms leveraging S3-compatible object storage, Apache Iceberg, Spark, and Trino.
• Build real-time streaming integration using Apache Kafka and Apache Flink for low-latency event processing, enrichment, and data distribution.
• Drive performance engineering and scalability across data ingestion, storage, streaming, query, and analytics platforms.
• Design and build self-service data platform capabilities including APIs, data products, metadata services, and developer enablement frameworks.
• Implement cloud-native platform engineering practices using Kubernetes/OpenShift, Terraform, CI/CD, GitOps, and Infrastructure as Code.
• Lead metadata, lineage, and data discovery capabilities using technologies such as DataHub, OpenMetadata, or Apache Atlas to improve platform transparency and usability.
• Ensure platform security and regulatory compliance through fine-grained access controls, encryption, secrets management, auditing, and secure-by-design engineering principles.

Bonus for knowledge of:

• Automation/configuration management using toolsets such as Puppet, Chef, Ansible or equivalent.
• Docker and Kubernetes.
• Knowledge and understanding of financial markets.
• Setting up CI/CD pipelines using Bamboo.
• Confluent certified developer for Apache Kafka.
• Dollar Universe.

Personal Qualities:

• Ability to work under pressure with changing priorities, with a view to resolving issues innovatively, and meeting key stakeholders expectations.
• A dynamic and self-motivated attitude.
• Accountable and proactive.
• Able to provide leadership and motivate team demonstrating strong interpersonal skills.
• Must display strong analytical skills and attention for detail.
• Demonstrates pragmatic judgement, balancing risk and business value to reach decisions which are well informed and actionable.