Principal Data Engineer in London

I am working with a scaling technology organisation operating across multiple brands within the property and legal technology space. As the business continues to grow and mature its data capability, they are now looking to hire a Principal Data Engineer to define and lead the technical direction of their data platform.

This is a senior, highly influential role sitting above multiple engineering teams. While there is a strong hands-on element, this position is first and foremost about technical leadership, architecture, and setting standards. You will shape how data is ingested, governed, and trusted across the organisation, playing a central role in building a robust and scalable data warehouse and data platform as the business evolves.

The environment spans Azure and AWS, with Databricks at the core of the data stack. You will work closely with senior engineering, product, and business stakeholders to design future-proof data architectures that can scale with increasing data volumes and complexity. As the organisation grows, this role offers a genuine opportunity to architect a modern data warehouse and lake that underpins insight across the UK housing market.

This role is ideal for a principal level engineer who enjoys operating at both strategic and execution levels. Someone comfortable defining architecture and governance with senior leaders, while also rolling up their sleeves to build and optimise complex data pipelines.

What you will be doing:

• Defining and owning the technical strategy for the data platform across ingestion, processing, and analytics
• Designing scalable, secure, and high-performance architectures using Databricks and distributed data systems
• Building and overseeing robust data ingestion pipelines using PySpark, with a strong focus on reliability and accuracy
• Ensuring end-to-end data quality from raw ingestion through to curated datasets used for reporting and analytics
• Establishing and enforcing best practices around data governance, lineage, metadata, and security, including Unity Catalogue
• Anticipating future scaling challenges and ensuring the platform is fit for long-term growth
• Acting as a technical authority across data engineering teams, raising standards and guiding architectural decisions
• Partnering with product, engineering, and commercial leaders to prioritise high-impact data initiatives
• Ensuring data platforms are compliant with GDPR and wider regulatory requirements
• Evaluating and introducing new technologies where they add clear value to the data ecosystem

What you will bring:

• Deep expertise in PySpark and distributed data processing at scale
• Strong hands-on experience designing, building, and optimising Databricks based platforms
• Advanced SQL skills including performance optimisation and schema design for analytical workloads
• Experience working across Azure based data warehouses, with some exposure to AWS
• Proven experience defining and leading complex data architectures rather than just implementing them
• Strong understanding of data governance, data quality frameworks, and security best practices
• A track record of influencing senior stakeholders and aligning data strategy with business objectives
• Experience mentoring senior engineers and setting technical standards without formal line management
• A strategic mindset with a clear focus on data reliability, scalability, and long-term business value

If you are a Principal Data Engineer looking for a role where you can genuinely shape a growing data platform and influence technical direction at scale, I would be happy to share further details in confidence.