Performance & Observability Engineer — AI-Driven Reliability

The Performance & Observability Engineer plays a critical role in ensuring system reliability, scalability, and visibility across the entire technology stack. This role focuses on transitioning from traditional monitoring to full observability, enabling deep performance insights, real-time issue detection, and proactive optimisation. The ideal candidate will have expertise in performance engineering, distributed tracing, logging, metrics, and automation, driving improvements across cloud, infrastructure, and application layers.

Primary Responsibilities

• Analyse application, database, and infrastructure performance to identify bottlenecks and inefficiencies.
• Develop performance benchmarks and SLIs to measure service responsiveness and stability.
• Collaborate with SRE and DevOps teams to optimise CI/CD pipelines for performance improvements.
• Collaborate with wider IT teams to Implement caching strategies, query optimisation, and autoscaling to enhance system efficiency.

Observability Platform Development & Implementation

• Design and implement end-to-end observability frameworks covering metrics, logs, traces, and events.
• Instrument services using existing tools (eg: Nexthink) to improve visibility.
• Enable distributed tracing across microservices to enhance root cause analysis and performance debugging.
• Standardise logging and telemetry collection across infrastructure, applications, and cloud services.
• Define best practices and consistent approach across development teams to improve monitoring consistency.

Maturing from Monitoring to Full Observability

• Transition from basic alerting to proactive insights, leveraging AI-driven anomaly detection.
• Ensure comprehensive observability across frontend, backend, databases, cloud infrastructure, and networking.
• Implement Service Level Indicators (SLIs), Service Level Objectives (SLOs), and Error Budgets to track system health.
• Automate root cause analysis and incident detection through advanced monitoring techniques.

Incident Response & Reliability Engineering

• Reduce Mean Time to Detect (MTTD) and Mean Time to Resolve (MTTR) through improved observability.
• Integrate monitoring and alerting tools with incident response platforms (ie: ServiceNow).
• Develop self-healing and auto-remediation mechanisms to reduce operational toil.
• Improve alerting strategies by reducing false positives and improving signal-to-noise ratio.

Governance, Compliance & Reporting

• Define observability best practices and governance models to ensure adoption across teams.
• Ensure log retention, security, and compliance with standards (e.g., GDPR, SOC 2, PCI DSS).
• Develop executive dashboards and reporting frameworks to showcase reliability and performance trends.

Key Performance Indicators

Maturity of Observability Capabilities

• % of Services with Full Observability Coverage - Ensure visibility across the entire stack.
• Instrumentation Completeness (%) - Track the number of services fully instrumented with logs, metrics, and traces.
• Service-Level Indicator (SLI) Coverage - Ensure key performance indicators are defined and tracked.
• Reduction in Blind Spots (%) - Improve monitoring coverage across all components.

Performance & Reliability Metrics

• Application Response Time (P99, P95, P50 Latency) - Improve service performance.
• System Throughput & Load Handling (%) - Increase service efficiency and scalability.
• Reduction in Performance Bottlenecks (%) - Optimise infrastructure and application layers.
• Successful Load Test Pass Rate (%) - Ensure applications meet expected performance benchmarks.
• Mean Time to Detect (MTTD) and Mean Time to Resolve (MTTR) - Reduce system downtime and recovery time.
• Reduction in Noisy Alerts (%) - Improve alert relevance and reduce false positives.
• Proactive Issue Detection (%) - Increase the percentage of incidents identified before user impact.
• Error Budget Utilization (%) - Ensure system reliability is balanced with innovation velocity.

Automation & AI-Driven Observability (AIOps)

• % of Issues Resolved via Automated Remediation - Reduce manual intervention in incident response.
• Reduction in On-Call Burden (%) - Minimize alerts requiring human intervention.
• Anomaly Detection Accuracy (%) - Improve proactive detection of performance issues.

Business & Customer Impact

• Customer Experience Metrics (Latency, Errors, Uptime) - Ensure observability drives tangible business improvements.
• Downtime Reduction (%) - Improve service availability and reliability.
• Cost Optimisation from Performance & Observability (%) - Reduce operational inefficiencies and cloud expenses.

Qualifications, Skills and Experience

Technical Skills

• Experience in using and maintaining Observability & APM Tools - Grafana experience is essential
• Experience of using KQL
• Performance Testing & Load Testing
• Cloud & Infrastructure Monitoring - AWS CloudWatch, Azure Monitor, GCP Operations Suite, Kubernetes Observability.
• Knowledge of Log Aggregation & Analysis - eg: Grafana Loki, Splunk etc
• Automation & Scripting - PowerAutomate, Terraform, PowerShell
• Sound understanding of firm's applications, systems and tools across technology stack
• DevOps experience is desirable
• Nexthink experience is desirable
• Strong problem-solving and root cause analysis skills.
• Ability to translate observability insights into business impact for stakeholders.
• A continuous improvement mindset, focused on reducing toil and improving efficiency.
• Experience working in a DevOps, SRE, or Platform Engineering environment.