Observability Engineer in City of Westminster

We're a hybrid business, which means you'll have flexibility alongside time together with your team.

In this role, you can expect at least two days per month in our London office, helping you stay connected, collaborate, and do your best work for our customers.

Note that there may be an expectation of other ad-hoc office days where necessary., Our SRE team is a high-trust, high-impact group of engineers who bring software engineering principles to operational reliability.

We are hands-on developers and systems thinkers who build scalable, observable, and resilient platforms.

Working closely with other Engineering, Data, and Platform & Operations (Plat Ops) teams, we help them build reliable, cost-efficient, and easy-to-diagnose systems and services.

We also help the Site Operations Engineering (SOE) team lead incident response, improve deployment safety, and guide teams towards a sustainable model of service ownership.

We collect, process, and export large volumes of telemetry data every day and are constantly evolving our approach to cost-efficiency.

Observability is a first-class concern that shapes how we build and support services across the business.

• Message From the
• Principal

Engineer: "Hi!

I'm Dan, the Principal Site Reliability Engineer here at Dunelm.

The SRE team has evolved quite a bit since I started three years ago.

We have moved from an engineer-per-team 'embedded' engagement model to a combination 'tools & consulting' model.

This has allowed us to scale our coverage more effectively and deliver more meaningful impact throughout Dunelm's service teams.

The other big change has been moving from proprietary instrumentation to Open Telemetry, giving us better flexibility, consistency, and control of the telemetry that we ingest, process, and export.

I'm looking forward to receiving your application and hopefully welcoming you into our team." Interview Process 1.

Introductory video call (around 45 minutes) with the Principal Engineer and Product Delivery Lead to get to know each other, explain the role, and hear about your experience, goals, and approach to work. 2.

Technical Discussion (90 minutes) with a few members of the SRE team.

You will work through scenario-based questions designed to help you to highlight your knowledge, specific approaches and where you feel any improvements could be made.

Important note: we understand the pressure people are under now in applying for multiple jobs and our straightforward two-stage process, without a practical aspect, is hopefully a refreshing change.

However, please read this job description carefully.

• 1.

Engineering & Automation Build automation and reliability tooling using well-architected, maintainable, testable, and easy to read & understand code.

Contribute to shared libraries, observability components, and internal platforms.

• 2.

Observability Instrumentation Help evolve our observability strategy across systems and services.

Drive how we collect, process, sample, and surface logs, distributed traces and metrics using Open Telemetry.

Focus on high-quality telemetry that enables fast diagnosis, cost efficiency, and meaningful visibility across the stack.

• 3.

Service Levels & Ownership Help teams truly own their services by helping to define and adopt meaningful SLIs and SLOs.

Guide product teams in using observability data to make reliability measurable.

• 4.

Incident Response Participate in on-call investigations when issues arise.

Drive blameless post-incident reviews and help to identify observability gaps, recommend mitigating actions that stem losses, but also permanent fixes that prevent recurrence.

• 5.

Infrastructure & Pipelines Model infrastructure using tools such as Pulumi, CDK, and Terraform in AWS and other Paa S and Saa S providers.

Improve pipelines and support safe deployment strategies (e. g. canary, blue / green).

• 6. Mentoring & Team Growth Help support and coach other engineers. Participate in technical discussions and share knowledge through pairing, planning, and documentation.
• 7.

Continuous Learning and Innovation Stay ahead of emerging practices in observability.

Lead team proof-of-concepts and introduce new patterns or tools that improve our platform.

• 8.

Strategic Development Contribute to the prioritisation of the SRE roadmap.

Help shape observability tooling, telemetry patterns, and platform-wide approaches to service ownership and reliability.

• 9.

Aligning to Business Goals Use observability insights to support product goals.

Help to ensure SRE priorities align with Dunelm's wider objectives for quality, performance, and customer experience.

At Dunelm, our shared values of Act Like Owners, Keep Listening & Learning, Long-Term Thinking, and Stronger Together serve as the foundation for our success.

These values guide us continuously; improving our practices and ensure we dedicate our time to what truly matters.

• As an Observability Engineer in the SRE team, you will exemplify these key behaviours:
• Support and build trust with teammates, always assuming positive intent
• Communicate clearly and share knowledge to build shared understanding
• Stay curious, ask why, and always look to improve how things work
• Embrace change, adapt quickly, and take on a variety of challenges
• Drive innovation by looking for better ways forward and pushing for progress

D&I statement Dunelm is committed to becoming an inclusive business that is representative of our customers and locations.

We recognise the value in diversity and welcome applications from all candidates regardless of age, race, religion, disability, gender or sexual identity, socio-economic background or education.

We are committed to making Dunelm a place where everyone can enjoy a successful career and have systems in place to support all our colleagues.

We are seeking applications from candidates who share our values and celebrate diversity in all its forms.

Our commitment to you Accessibility allows everyone to have an equitable chance when applying and showing off their potential, which helps us create a more inclusive and diverse team that brings fresh ideas and perspectives.

• Software Engineering: Languages
• Most of our services are in Type Script, Rust, and Python, so knowledge of at least one of these is required.

However, deep knowledge of specific programming languages is not as important as understanding what the code is doing (or understanding what AI tells you the code is doing).

• Software Engineering: Principles
• We value idiomatic, pragmatic, and testable code with strong, appropriate automated testing.

How that code is produced (using AI and / or manually) is not as important as you being able to explain (and potentially defend it) in a code review.

• Architecture
• Most of our services run on AWS and are serverless (Lambda) or containerised (ECS & Fargate), so specific knowledge of deploying and running software in those environments, plus general networking principles, is required.
• Open Telemetry