Performance & Observability Engineer

The Performance & Observability Engineer plays a critical role in ensuring system reliability, scalability, and visibility across the entire technology stack.

This role focuses on transitioning from traditional monitoring to full observability, enabling deep performance insights, real-time issue detection, and proactive optimisation.

The ideal candidate will have expertise in performance engineering, distributed tracing, logging, metrics, and automation, driving improvements across cloud, infrastructure, and application layers.

• Primary Responsibilities
• Analyse application, database, and infrastructure performance to identify bottlenecks and inefficiencies.
• Develop performance benchmarks and SLIs to measure service responsiveness and stability.
• Collaborate with SRE and Dev Ops teams to optimise CI/CD pipelines for performance improvements.
• Collaborate with wider IT teams to Implement caching strategies, query optimisation, and autoscaling to enhance system efficiency.
• Observability Platform Development & Implementation
• Design and implement end-to-end observability frameworks covering metrics, logs, traces, and events.
• Instrument services using existing tools (eg: Nexthink) to improve visibility.
• Enable distributed tracing across microservices to enhance root cause analysis and performance debugging.
• Standardise logging and telemetry collection across infrastructure, applications, and cloud services.
• Define best practices and consistent approach across development teams to improve monitoring consistency.
• Maturing from Monitoring to Full Observability
• Transition from basic alerting to proactive insights, leveraging AI-driven anomaly detection.
• Ensure comprehensive observability across frontend, backend, databases, cloud infrastructure, and networking.
• Implement Service Level Indicators (SLIs), Service Level Objectives (SLOs), and Error Budgets to track system health.
• Automate root cause analysis and incident detection through advanced monitoring techniques.
• Incident Response & Reliability Engineering
• Reduce Mean Time to Detect (MTTD) and Mean Time to Resolve (MTTR) through improved observability.
• Integrate monitoring and alerting tools with incident response platforms (ie: Service Now).
• Develop self-healing and auto-remediation mechanisms to reduce operational toil.
• Improve alerting strategies by reducing false positives and improving signal-to-noise ratio.
• Governance, Compliance & Reporting
• Define observability best practices and governance models to ensure adoption across teams.
• Ensure log retention, security, and compliance with standards (e. g., GDPR, SOC 2, PCI DSS).
• Develop executive dashboards and reporting frameworks to showcase reliability and performance trends.
• Key Performance Indicators
• Maturity of Observability Capabilities
• % of Services with Full Observability Coverage - Ensure visibility across the entire stack.
• Instrumentation Completeness (%) - Track the number of services fully instrumented with logs, metrics, and traces.
• Service-Level Indicator (SLI) Coverage - Ensure key performance indicators are defined and tracked.
• Reduction in Blind Spots (%) - Improve monitoring coverage across all components.
• Performance & Reliability Metrics
• Application Response Time (P99, P95, P50 Latency) - Improve service performance.
• System Throughput & Load Handling (%) - Increase service efficiency and scalability.
• Reduction in Performance Bottlenecks (%) - Optimise infrastructure and application layers.
• Successful Load Test Pass Rate (%) - Ensure applications meet expected performance benchmarks.
• Mean Time to Detect (MTTD) and Mean Time to Resolve (MTTR) - Reduce system downtime and recovery time.
• Reduction in Noisy Alerts (%) - Improve alert relevance and reduce false positives.
• Proactive Issue Detection (%) - Increase the percentage of incidents identified before user impact.
• Error Budget Utilization (%) - Ensure system reliability is balanced with innovation velocity.
• Automation & AI-Driven Observability (AIOps)
• % of Issues Resolved via Automated Remediation - Reduce manual intervention in incident response.
• Reduction in On-Call Burden (%) - Minimize alerts requiring human intervention.
• Anomaly Detection Accuracy (%) - Improve proactive detection of performance issues.
• Business & Customer Impact
• Customer Experience Metrics (Latency, Errors, Uptime) - Ensure observability drives tangible business improvements.
• Downtime Reduction (%) - Improve service availability and reliability.
• Cost Optimisation from Performance & Observability (%) - Reduce operational inefficiencies and cloud expenses.

Qualifications, Skills and Experience

• Technical Skills
• Experience in using and maintaining Observability & APM Tools - Grafana experience is essential
• Experience of using KQL
• Performance Testing & Load Testing
• Cloud & Infrastructure Monitoring - AWS Cloud Watch, Azure Monitor, GCP Operations Suite, Kubernetes Observability.
• Knowledge of Log Aggregation & Analysis - eg: Grafana Loki, Splunk etc
• Automation & Scripting - Power Automate, Terraform, Power Shell
• Sound understanding of firm's applications, systems and tools across technology stack
• Dev Ops experience is desirable
• Nexthink experience is desirable
• Strong problem-solving and root cause analysis skills.
• Ability to translate observability insights into business impact for stakeholders.
• A continuous improvement mindset, focused on reducing toil and improving efficiency.
• Experience working in a Dev Ops, SRE, or Platform Engineering environment.
• #J-18808-Ljbffr