The UK Health Security Agency (UKHSA) prevents, prepares for and responds to infectious diseases, and environmental hazards, to keep all our communities safe, save lives and protect livelihoods.
We provide scientific and operational leadership, working with local, national and international partners to protect the public’s health and build the nation’s health security capability.
UKHSA is an executive agency, sponsored by the Department of Health and Social Care.
Job overview
An SRE engineer will apply engineering principles to remediate infrastructure and operational problems. The primary focus will be on automation and CI/CD; ensuring our services run reliably, are scalable, and perform optimally in production environments. The role will monitor and manage these aspects while taking responsibility for multiple cloud infrastructure services. Observability of systems will be key to prioritising the operational service improvements and performance improvements to meet and exceed SLOs (Service Level Objectives).
Advert
Working with the HPC & SRE Team to:
- Ensure services are stable, scalable, performant and automated
- Respond to incidents, troubleshooting issues, and restoring services as quickly as possible
- Prioritise operational service improvements to meet or increase SLO, minimising downtime
- Ensure that effective monitoring/alerting is in place to proactively identify issues using tools and dashboards. Reducing times to respond to issues.
- Leverage automation to streamline tasks, reduce overhead on repeatable operations, reduce manual intervention and improve efficiency. Write code that is maintainable, clear, and concise
- Optimise system performance using strong problem-solving skills to identify bottlenecks with an engineering mindset
- Ensure systems can handle current and future workloads through automation and capacity planning
- Continuously improve services through observability, and identify ways to improve observability practices
- Follow SRE principles. Guide and educate stakeholders to adopt implemented principles
- Provide technical documentation for engineers. Providing training, where appropriate
- Working closely with engineering and technology teams to improve operational processes, reduce manual tasks, ensure seamless collaboration/knowledge sharing, reduce risks and adapt to new ways of working.
Working for our organisation
We pride ourselves as being an employer of choice, where Everyone Matters promoting equality of opportunity to actively encourage applications from everyone, including groups currently underrepresented in our workforce. UKHSA ethos is to be an inclusive organisation for all our staff and stakeholders. To create, nurture and sustain an inclusive culture, where differences drive innovative solutions to meet the needs of our workforce and wider communities. We do this through celebrating and protecting differences by removing barriers and promoting equity and equality of opportunity for all. Please visit our careers site for more information https://gov.uk/ukhsa/careers
Detailed job description and main responsibilities
We are seeking a highly motivated and experienced Site Reliability Engineer (SRE) to join our HPC & SRE engineering team. As an SRE, you will play a critical role in ensuring the stability, scalability, and performance of our services. You will combine software engineering and systems engineering to build, improve and run reliable, scalable production systems.
Key Responsibilities:
Service Reliability & Performance
- Ensure services are stable, scalable, and performant through engineering best practices and system design.
- Proactively identify and address system bottlenecks using advanced problem-solving and performance tuning techniques.
- Conduct capacity planning and implement solutions to ensure systems can support current and future workloads.
Incident Response & Troubleshooting
- Respond swiftly to production incidents, ensuring minimal downtime and quick restoration of services.
- Perform root cause analysis and postmortems, implementing lessons learned to prevent recurrence.
Monitoring, Alerting & Observability
- Contribute to the design and implementation of effective monitoring and alerting systems using tools and dashboards.
- Improve observability of services, ensuring issues are identified and addressed before impacting users.
- Continuously refine monitoring practices to reduce alert fatigue and improve response times.
Automation & Tooling
- Develop automation to eliminate manual, repetitive tasks and improve operational efficiency.
- Write clear, maintainable, and well-tested code to support automation efforts and system tooling.
- Drive initiatives to reduce operational toil and improve reliability through Infrastructure as Code (IaC).
Service Level Objectives (SLOs) & Operational Improvements
- Contribute to the definition, tracking, and continuous improvement of SLOs, SLIs, and error budgets.
- Identify and prioritize operational improvements that align with business goals and user experience.
SRE Best Practices & Advocacy
- Helping to evangelize SRE principles across the organization.
- Collaborate with stakeholders to integrate reliability practices into the development lifecycle.
Collaboration & Knowledge Sharing
- Work closely with software engineering, DevOps, and infrastructure teams to streamline deployment and operational workflows.
- Improve cross-functional collaboration and promote a culture of shared responsibility for service reliability.
Documentation & Training
- Maintain accurate technical documentation, runbooks, and post-incident reports.
- Provide training and mentorship to engineering teams on best practices and tools.
This list is not exhaustive .
Essential Criteria:
- Experience as a Site Reliability Engineer, DevOps Engineer, Operations Engineer or similar role
- Coding skills in programming/scripting languages such as Python, PowerShell or Bash
- Understanding of Linux/Unix & Windows systems, networking, and distributed systems
- Experience with observability tools (e.g., Prometheus, Grafana, Datadog) and alerting systems
- Understanding of infrastructure automation (e.g., Terraform, Ansible, PowerShell, Helm)
- Excellent communication and collaboration skills
- Possesses problem solving skills and the ability to respond to sudden unexpected demands
Desirable criteria:
- Experience with CI/CD pipelines, cloud platforms (e.g., AWS, GCP, Azure) and container orchestration (e.g., Kubernetes)
- Experience with post-incident reviews
- Previous involvement in driving adoption of SRE practices across an organization
- Experience delivering training or mentoring junior engineers
Selection Process Details
This vacancy is using Success Profiles and will assess your Behaviours, Experience and Technical skills.
Stage 1: Application & Sift
Success Profiles - GOV.UK
You will be required to complete an application form. You will be assessed on the listed 7 essential criteria, and this will be in the form of a:
- Application form (‘Employer/ Activity history’ section on the application)
-
1000 word supporting statement.
This should outline how you consider your skills, experience and knowledge provide evidence of your suitability for the role, with reference to the essential criteria.
You will receive a joint score for your application form and statement. (The application form is the kind of information you would put into your C.V –please be advised you will not be able to upload your CV. Please complete the application form in as much detail as possible). Please do not email us your CV.
Longlisting:
In the event of a large number of applications we will longlist into 3 piles of:
- Meets all essential criteria
-
Meets some essential criteria
- Meets no essential criteria
If used, the pile(s) ‘Meets all essential criteria’ will proceed to shortlisting.
Shortlisting:
In the event of a large number of applications we may conduct an initial sift, on the lead criteria of:
- Experience as a Site Reliability Engineer, DevOps Engineer, Operations Engineer or similar role
Desirable criteria may be used in the event of a large number of applications/large amount of successful candidates.
If you are successful at this stage, you will progress to interview & assessment.
Feedback will not be provided at this stage.
Stage 2: Interview
Success Profiles - GOV.UK
You will be invited to a face to face interview at Canary Wharf. London.
If face to face interviews are planned, in exceptional circumstances, we may be able to offer a remote interview.
Behaviours and technical skills will be tested at interview through questioning and a presentation.
There will be a Presentation required, the subject being:
Automating a complex operational process.
The Behaviours tested during the interview stage will be:
- Changing and improving. (Lead behaviour)
-
Working together.
-
Managing a quality service.
- Delivering at pace.
Interview dates are to be confirmed.
Once this job has closed, the job advert will no longer be available. You may want to save a copy for your records.
Location
This role is being offered as hybrid working based at one of our Core HQ's or Scientific Campuses.
We offer great flexible working opportunities at UKHSA and operate using a hybrid working model where business needs allow. This provides us with greater flexibility about how and where we work, to get the best from our workforce. As a hybrid worker, you will be expected to spend a minimum of 60% of your contractual working hours (approximately 3 days a week pro rata, (averaged over a month) working at one of UKHSA's core HQs (Birmingham, Leeds, Liverpool, and London), or scientific campus sites (Colindale, Porton or Chilton).
Our core HQ offices are modern and newly refurbished with excellent city centre transport links and benefit from co-location with other government departments such as the Department for Health and Social Care (DHSC).
Security Clearance Level Requirement
Successful candidates must pass a basic disclosure and barring security check before they can be appointed.
Successful candidates must meet the security requirements before they can be appointed. The level of security needed is Security Check.
For meaningful National Security Vetting checks to be carried out individuals need to have lived in the UK for a sufficient period of time. You should normally have been resident in the United Kingdom for the last 5 years as the role requires Security Check (SC) clearance. UK residency less than the outlined periods may not necessarily bar you from gaining national security vetting and applicants should contact the Vacancy Holder/Recruiting Manager listed in the advert for further advice.
Successful candidates must meet the security requirements before they can be appointed.
Eligibility Criteria
External:This vacancy is open to all external applicants (anyone) from outside the Civil
Service as well as internal applicants.
Salary Information
If you are successful at interview, and are moving from another government department, NHS, or Local Authority, the relevant starting salary principles for level transfers or promotions will apply. Otherwise, roles are offered at the pay scale minimum for the grade, but in exceptional circumstances there may be flexibility if you are able to demonstrate you are already in receipt of an existing, higher salary. Pay increases are through the relevant annual pay award for the role and terms.
Please be aware that the salary is based on the office location.
Senior Executive Officer (SEO)
- £41,983- £48,128 (National) (+MPS of up to £5,000 per annum, pro rata)
- £44,148- £50,121 (Outer London) (+MPS of up to £5,000 per annum, pro rata)
- £46,310- £52,113 (Inner London) (+MPS of up to £5,000 per annum, pro rata)
Market Pay Supplement (MPS) is offered for this role of up to £5,000 per annum, pro rata. This is due to be reviewed 31/3/27. The amount offered in MPS will be dependant on the postholder's capability level. Each post holder would be assessed on entry to the civil service with the majority being placed on the lowest award unless clear evidence can be provided of higher capability level. The MPS bands are:
- Developing - up to £2,500 per annum, pro rata
- Proficient - up to £3,500 per annum, pro rata
- Accomplished - up to £5,000 per annum, pro rata
IMPORTANT - PLEASE READ
If you experience technical difficulties when using Trac, please contact Support via https://apps.trac.jobs/support and provide screenshots/details of the issue encountered. If Trac cannot resolve your issue, you should notify the UKHSA Recruitment Team at [email protected] as soon as possible.
Please note that where no technical issue has been recorded by Trac, and no supporting evidence has been provided before the closing deadline, we will be unable to accept late applications. We therefore recommend submitting applications well in advance of the closing date and taking screenshots of any issues encountered.
Qualifications and Registrations
For roles where specific qualifications or registrations are required, successful applicants will be asked to provide appropriate evidence. Employment cannot commence until satisfactory documentation has been received and verified.
Artificial Intelligence (AI)
Artificial Intelligence can be a useful tool to support your application, however, all examples and statements provided must be truthful, factually accurate and taken directly from your own experience. Where plagiarism has been identified (presenting the ideas and experiences of others, or generated by artificial intelligence, as your own) applications may be withdrawn and internal candidates may be subject to disciplinary action. Please see our candidate guidance for more information on appropriate and inappropriate use.
Link below:
Artificial intelligence and recruitment | Civil Service Careers
This is a Non-Reserved post under the Civil Service Nationality Rules. To be eligible for employment in the UK Civil Service applicants must meet the Civil Service Nationality Rules (CSNRs) which operate independently of and additionally to the Immigration Rules. Applicants must also meet necessary security and vetting requirements, along with any other relevant pre-employment checks.
This job is broadly open to the following groups:
- UK nationals
- nationals of the Republic of Ireland
- nationals of Commonwealth countries who have the right to work in the UK
- nationals of the EU, Switzerland, Norway, Iceland or Liechtenstein and family members of those nationalities with settled or pre-settled status under the European Union Settlement Scheme (EUSS) (opens in a new window)
- nationals of the EU, Switzerland, Norway, Iceland or Liechtenstein and family members of those nationalities who have made a valid application for settled or pre-settled status under the European Union Settlement Scheme (EUSS)
- individuals with limited leave to remain or indefinite leave to remain who were eligible to apply for EUSS on or before 31 December 2020
- Turkish nationals, and certain family members of Turkish nationals, who have accrued the right to work in the Civil Service
For more information on job nationality requirements and the right to work in the UK, see the Civil Service Nationality rules (opens in a new window) and the UK Visas and Immigration rules (opens in a new window)
For posts on UKHSA Civil Service terms and conditions, new entrants to the Civil Service are expected to start on the minimum of the pay band. For existing Civil Servants and roles advertised across government, the rules of transfer apply, i.e., level transfers move on current salary or the pay range minimum, transfers on promotion move to new pay range minimum or receive a 10% increase. Either case is determined by whichever is the highest.
The Civil Service pay structure and progression is different from NHS Agenda for Change (AfC), most local authority pay grades and other systems that have annual pay increments. For further details, please refer to the Information Sheet- Starting Salaries & Benefits attachment.
For AfC or Medical/Dental posts, you must have the correct professional registration to be appointed. The pay will follow the AfC or Medical & Dental terms & conditions. You may be asked to provide evidence of previous service whilst we are conducting pre-employment checks to determine your starting salary.
For Temporary Appointments, if you are not currently a civil servant, you will take up the post on a Fixed Term appointment. You may be able to take this role up as a Secondment. If you are an existing Civil Servant, based outside of the UKHSA, you will take up the post as a loan which you will need your department to agree. You cannot take the post up as a fixed term. If you are an existing UKHSA member of staff, you will take up the post as either a level transfer or a temporary promotion as per the UKHSA’s Pay policy.
From time to time, the role holder may be required to support UKHSA’s incident response activity by contributing their appropriate expertise within an Incident Cell; during such periods, supported by the line manager, this work will take priority over normal day‑to‑day responsibilities, commensurate with their grade, skills and experience.
Late Applications will unfortunately not be considered.
Working for the Civil Service
We recruit by merit on the basis of fair and open competition, as outlined in the Civil Service Commission's recruitment principles (opens in a new window).
The Civil Service embraces diversity and promotes equality of opportunity. The law requires that selection for appointment to the Civil Service is on merit on the basis of fair and open competition, as outlined in the Civil Service Commission's Recruitment Principles.
If you feel your application has not been treated in accordance with the Recruitment Principles, and you wish to make a complaint, in the first instance, you should contact UKHSA Public Accountability Unit via email: [email protected] If you are not satisfied with the response you receive from the Department, you can contact the Civil Service Commission: Visit the Civil Service Commission website here.
Reserve List - If more than the required number of suitable candidates pass the interview criteria, you may be kept on a reserve list for 12 months subject to your agreement. You may be contacted, in merit-order, if similar roles with closely matching essential criteria become available and the department choose to appoint from a reserve list.
The panel will assess if candidates meet the requirement of the role first, using a specific benchmark system. If you are interviewed for the post and do not meet the required threshold for the specified grade, your application may be assessed against a similar, lower grade role and you may be offered the post should one be available.
Interview expenses will not be reimbursed.
UKHSA is required to check employment and/or education history covering three consecutive years. Please ensure you give details of at least two different referees, even if you were employed in one company for three years or more.
If you are offered a job, information will also be transferred into the national NHS Electronic Staff Records system. Please note, all communication regarding your application will be made via email, please ensure you check your junk/spam folders as emails are sometimes filtered there.
Any move to UKHSA from another employer will mean you can no longer access childcare vouchers. This includes moves between government departments. You may however be eligible for other government schemes, including Tax-Free Childcare. Determine your eligibility at https://www.childcarechoices.gov.uk/.
Benefits of working at UKHSA include:
- Generous annual leave:
-26 days for the first 5 years of continuous service
-28 days after 5 years of continuous service
-32 days after 10 years of continuous service
-Plus public holidays and one privilege day for the King’s birthday - Access to a generous Defined Benefit pension scheme with employer contributions.
- Access to a cycle-to-work salary sacrifice scheme, season ticket advances and payroll giving.
- Access to a retail discounts and cashback site.
- We also promote flexible working patterns (part-time, job-share, condensed hours). UKHSA views flexible working as essential in enabling us to recruit and retain talented people, ensuring that they are able to enjoy a long-lasting career with us. All employees have the right to apply for flexible working and there are a range of options available including working from home, compressed hours and job sharing.
- We also offer a generous maternity/ paternity and adoption leave package.
Hybrid Working
UKHSA operates a hybrid working model where business needs allow. This provides us with greater flexibility about how and where we work, to get the best from our workforce.
As a hybrid worker, you will usually spend a minimum of 60% of your contracted hours (averaged over a month) working at one of UKHSA's locations (approximately 3 days a week pro rata) and the rest of your time working from home.
Disability Confident Scheme
The Civil Service embraces diversity and promotes equal opportunities. As such, we run a Disability Confident Scheme for candidates with disabilities who meet the minimum selection criteria at sift to ensure these candidates are invited to interview. If you wish to be included in this scheme please tick the box on your application form.
Reasonable Adjustments
The Civil Service is committed to making sure that our selection methods are fair to everyone.
To help you during the recruitment process, we will take into account any reasonable adjustments that could help you.
An adjustment is a change to the recruitment process or an adjustment at work.
This is separate to the Disability Confident Scheme.
If you need an adjustment to be made at any point during the recruitment process you should:
Contact the recruitment team in confidence as soon as possible to discuss your needs.
You can find out more information about reasonable adjustments across the Civil Service here: https://www.civil-service-careers.gov.uk/reasonable-adjustments/
International Police Check
If you have spent more than 6 months abroad over the last 3 years you may need an International Police Check. This would not necessarily have to be in a single block, and could be time accrued over that period.
Internal Fraud check
If successful for this role as one aspect of pre-employment screening, applicant’s personal details – name, national insurance number and date of birth - will be checked against the Cabinet Office Internal Fraud Hub and anyone included on the database will be refused employment unless they can show exceptional circumstances. Currently this is only for External candidates to the Civil Service.
Security Vetting
Please check the Security Clearance needed for the role and follow the link for more information:
https://www.gov.uk/government/publications/united-kingdom-security-vetting-clearance-levels/national-security-vetting-clearance-levels
Future location
UKHSA is investing in a new state-of-the-art National Biosecurity Centre in Harlow, Essex, which will eventually bring together teams currently based at Canary Wharf, Colindale and Porton Down. For more details, please see: Huge biosecurity centre investment to boost pandemic protection - GOV.UK.
The new facilities will start becoming operational in the mid-2030s, with full completion by 2038. Staff will move in phases as facilities become available. If you're appointed to a role currently based at Canary Wharf, Colindale or Porton Down, please note that we'll continue investing in these sites for the next decade. As we get closer to the transition, we'll provide full information about relocation support available to staff.
Agenda for Change roles -NHS Pension
For roles offered under Agenda for Change (AFC) Terms and Conditions (and if you are eligible)
• If you are already on AFC T&Cs or accept the role on a fixed-term contract, you will be enrolled in the NHS Pension Scheme or keep your current AFC Pension.
• If you take the role on a loan or temporary promotion, you will retain your existing, substantive T&Cs.
• If the role is available under either AFC or Civil Service T&Cs depending on your registration, your pension scheme will align accordingly.
Senior Specialist Engineer (Specialist Site Reliability Engineer SRE) in London employer: UK Health Security Agency
At UKHSA, we pride ourselves on being an exceptional employer, offering a collaborative and inclusive work culture that values innovation and professional growth. As a Lead Cyber Security Architect, you will have the opportunity to make a significant impact on public health security while enjoying competitive salaries, including a market pay supplement, and the flexibility of remote work. Our commitment to employee development ensures that you will continuously enhance your skills in a supportive environment dedicated to safeguarding national services.
