Senior Platform Engineer - Observability

Teya is hiring a Senior Platform Engineer to join the Observability team within Cloud Platform. This team builds and operates the observability products and foundations used by engineering across Teya. The role is not a traditional operations or support position. It is a senior engineering role focused on building reliable, scalable and self-service observability capabilities that help hundreds of teams move faster while improving reliability, performance and visibility. You will work on systems with meaningful scale and company-wide impact. Cloud Platform supports self-service software delivery across the organisation, and the observability stack underpins monitoring, alerting, logging, tracing and dashboards used by teams across multiple scopes and environments.

What this team owns

• The platform currently operates at large scale, including 13 monitoring stacks, hundreds of thousands of metric samples per second, tens of thousands of traces per second, and significant long-term storage footprints across metrics, logs and traces.
• The Observability team is responsible for core platform capabilities across:
• Monitoring built around Prometheus, Thanos, exporters, recording rules and alerting.
• Logging built around Loki and associated querying, storage and alerting workflows.
• Tracing built around Tempo and OpenTelemetry-compatible ingestion paths.
• Visualisation and user workflows through Grafana.
• Self-service observability for product teams, including metric collection, dashboards, recording rules, alerting definitions and integrations with Slack and incident tooling.

Responsibilities

• Design, build, deploy and improve observability products used across Cloud Platform and engineering teams.
• Improve the scalability, reliability and performance of monitoring, logging, tracing and alerting systems.
• Build self-service abstractions that let teams instrument services, define alerts, create dashboards and troubleshoot systems with minimal platform dependency.
• Help define and evangelise best practices around observability, reliability and performance engineering.
• Work closely with engineers across the company to improve telemetry quality, reduce toil and raise the standard of operational excellence.
• Contribute to platform-wide engineering work in areas such as automation, Kubernetes, cloud infrastructure, incident learning and internal tooling.

What we are looking for

• Strong software engineering background, ideally with significant experience in Golang and the broader software development lifecycle.
• Strong understanding of distributed systems, reliability concerns and performance trade-offs at scale.
• Hands‑on experience with modern observability tooling such as Prometheus, Grafana, Loki, Tempo, Thanos, OpenTelemetry or similar systems.
• Experience with cloud platforms and their APIs, plus infrastructure patterns in environments such as AWS.
• Experience working with Kubernetes and container‑based platforms.
• Ability to operate with high ownership in a senior team and make sound engineering decisions in systems with large organisational blast radius.
• Deep understanding of PromQL and LogQL.
• Good intuition for latency, tail latency, sampling, percentiles and telemetry trade-offs between logs, metrics and traces.
• Experience designing or operating observability systems for many teams, not only for a single service or product.
• Experience improving developer experience through automation and platform abstractions.

The Perks

• We trust you, so we offer flexible working hours, as long it suits both you and your team.
• Physical and mental health support through our partnership with GymPass giving free access to over 1,500 gyms in the UK, 1‑1 therapy, meditation sessions, digital fitness and nutrition apps.
• Our company offers extended and improved maternity and paternity leave choices, giving employees more flexibility and support.
• Cycle‑to‑Work Scheme.
• Health and Life Insurance.
• Pension Scheme.
• 25 days of Annual Leave (+ Bank Holidays).
• Office snacks every day.
• Friendly, comfortable and informal office environment in Central London.

Teya is proud to be an equal opportunity employer. We are committed to creating an inclusive environment where everyone regardless of race, ethnicity, gender identity or expression, sexual orientation, age, disability, religion, or background can thrive and do their best work. We believe that a diverse team leads to better ideas, stronger outcomes, and a more supportive workplace for all. If you require any reasonable adjustments at any stage of the recruitment process whether for interviews, assessments, or other parts of the application—we encourage you to let us know. We are committed to ensuring that every candidate has a fair and accessible experience with us.