Software Engineer (Prometheus / Grafana) in Cardiff

SRT Marine Systems plc (SRT) is a market leader in the domain of international marine surveillance technology and systems. We are a respected, established, and ambitious multi‑national company headquartered in the UK with a global customer base. The company has a worldwide impact in the marine sector by leading the next generation of maritime domain awareness technologies (MDA), products, and systems that significantly enhance security, safety, environmental protection, and sustainability. Our customers range from the largest national coast guards to individual vessel owners.

As a Software Engineer (Prometheus / Grafana) at SRT, you will be part of a small team tasked with implementing an end‑user observability visualisation. Currently, we have observability dashboards in place for our engineers, utilising Prometheus for metrics collection and Grafana for visualisation. This initiative aims to deliver a more user‑friendly solution tailored for our end‑users. Our clients are located across various countries worldwide, each with differing WAN capabilities, and our system is geographically distributed on‑premises across multiple sites. We are fortunate to have a team of highly experienced engineers, including UX designers, who can provide support and guidance.

Key Responsibilities:

• Monitoring & Metrics Collection: Design, configure, and maintain Prometheus‑based monitoring solutions; Develop and manage metric exporters for application and system‑level data; Optimise Prometheus scraping configurations and retention policies.
• Alerting & Incident Response: Define and maintain alert rules based on SLIs/SLOs and performance baselines; Ensure alerts are actionable, with minimal false positives; Participate in on‑call rotations and incident postmortems.
• Observability Dashboards: Design and maintain Grafana dashboards for real‑time operational insights; Collaborate with engineering and product teams to create tailored visualisations; Provide self‑service dashboard capabilities for end users.
• System Performance & Reliability: Monitor infrastructure (servers, containers, databases, services) for uptime, latency, and throughput; Identify bottlenecks and recommend improvements.

Required Skills & Experience:

• Proven experience with Prometheus (including PromQL) and Grafana in production environments.
• Strong knowledge of Linux‑based systems.
• Experience writing and optimising PromQL queries for alerts and dashboards.
• Familiarity with exporters (node_exporter, blackbox_exporter, custom exporters).
• Understanding of alertmanager configuration and routing.
• Proficiency with Grafana dashboard creation and templating.
• Strong troubleshooting skills for infrastructure and application issues.
• Familiarity with containers (Docker).
• Scripting skills with a focus on Python (Bash or Go also beneficial) for automation.

Please note – you will be required to come to our Cardiff office 1 day a week.

Benefits:

• Highly Competitive Salary & Benefits package.
• Matched company pension contributions up to 5%.
• 25 days annual leave rising to 28 days with service.
• Career development opportunities.

SRT Marine Systems plc are an equal opportunity employer. We are committed to creating an inclusive working environment for all employees and actively encourage applications from all sectors of the community.