Lead DevOps in London

DescriptionMethods is a 100M+ IT Services Consultancy who has partnered with a range of central government departments and agencies to transform the way the public sector operates in the UK. Established over 30 years ago and UK-based, we apply our skills in transformation, delivery, and collaboration from across the Methods Group, to create end-to-end business and technical solutions that are people-centred, safe, and designed for the future.Our human touch sets us apart from other consultancies, system integrators and software houses - with people, technology, and data at the heart of who we are, we believe in creating value and sustainability through everything we do for our clients, staff, communities, and the planet.We support our clients in the success of their projects while working collaboratively to share skill sets and solve problems. At Methods we have fun while working hard; we are not afraid of making mistakes and learning from them.Methods was acquired by the Alten Group in early 2022.The role is heavily AWS platform engineering focused. While Azure and DevSecOps experience are valuable, the day-to-day work is centred on designing, maintaining and improving AWS-hosted Identity platforms running on Kubernetes and ECS.This is a hands-on technical leadership role. You will be expected to lead by example, mentor engineers, influence technical decisions and own services throughout their lifecycle.RequirementsCore Technical SkillsAWSExpected to have strong hands-on experience across AWS services including:ECSEKSEC2IAMAuto Scaling GroupsKMSCloudHSMSecrets ManagerSystems Manager Parameter StoreCloudWatchThis is an engineering role rather than an infrastructure administration role.Infrastructure as CodeStrong Terraform experience is expected.This includes:Writing production-quality TerraformCreating and maintaining reusable Terraform modulesManaging multiple environments using Terraform WorkspacesReviewing and improving existing modulesFollowing engineering standards and code review practicesKubernetesA high level of Kubernetes knowledge is expected.Daily work includes:Deploying and operating applications on Amazon EKSTroubleshooting workloadsWorking with Deployments, Services, Ingress, ConfigMaps and SecretsUnderstanding RBACResource requests and limitsHorizontal Pod AutoscalingNetwork PoliciesKubernetes troubleshootingHelmHelm is used extensively.You will be expected to:Write Helm chartsMaintain existing chartsBuild reusable templatesManage values across environmentsDebug Helm deploymentsThis is a core skill for the role.ECSA significant amount of the estate still runs on ECS.Expected knowledge includes:ECS ServicesTask DefinitionsAuto Scaling GroupsCapacity ProvidersDeployment strategiesOperational troubleshootingThe team owns the ECS platform.DockerAlmost every application is containerised.Expected experience:Building imagesWriting DockerfilesMulti-stage buildsImage optimisationContainer troubleshootingGitLab & CI/CDAll source code is hosted in GitLab.CI/CD is implemented using GitLab CI.Expected experience includes:Writing GitLab CI pipelinesMaintaining reusable pipeline fragments and templatesBuilding and maintaining reusable CI/CD componentsManaging multi-stage deployment pipelines across Development, Test, Stage and ProductionPipeline troubleshootingSecure secret handlingBuild and deployment automationTesting & Quality EngineeringTesting is considered part of delivering a feature, not an optional activity.Lead Engineers are expected to understand how to test infrastructure, platform code and applications, and to integrate those tests into automated delivery pipelines.Expected experience includes:Writing and maintaining Terraform TestsWriting and maintaining Go-based tests for Helm charts where appropriateWriting or extending application tests (for example Node.js unit or integration tests)Writing automated tests for Python tooling and applications using frameworks such as pytestUnderstanding test coverage and identifying gapsIntegrating automated tests into GitLab CI pipelinesTroubleshooting failing automated testsBuilding reusable testing stages within GitLab CI templatesRunning tests locally and within CI/CD pipelinesPerformance, load and stress testing using JMeter and k6Integrating performance testing into GitLab CI pipelines where appropriateAnalysing performance test results and identifying bottlenecksProducing repeatable automated validation rather than relying on manual testingTesting is expected to form part of the Definition of Done.Identity & SecurityAs an Identity programme, understanding authentication is important.Expected knowledge includes:OAuth 2.0 flowsOpenID Connect (OIDC)JWTsAPI authenticationIAMKubernetes RBACKnowledge of identity platforms and authentication patterns is highly desirable.API GatewayThe platform makes extensive use of Kong API Gateway.Useful experience includes:RoutingAuthentication pluginsOIDC integrationRate limitingUpstream configurationTroubleshooting gateway behaviourService MeshApplications running on EKS use Istio.Expected knowledge includes:Virtual ServicesDestination RulesGatewaysmTLSTraffic routingBasic troubleshootingMonitoring & ObservabilityCurrent monitoring stack:PrometheusGrafanaThe organisation is introducing Dynatrace, either alongside or as a future replacement.Expected experience includes:Creating dashboardsAlertingMetricsPerformance troubleshootingRoot cause analysisMonitoring production servicesTechnical DocumentationStrong technical documentation is an expected part of the role.You will be expected to:Produce clear Spike documentation covering the problem, investigation, options considered, recommendation and outcomeWrite implementation documentation for new features and architectural changesMaintain comprehensive README files for repositories, Terraform modules and Helm chartsProduce operational runbooksCreate support documentationProduce handover documentation where appropriateRecord technical decisions and their reasoningCommunicate ideas and proposals in a structured wayWrite high-quality Jira stories, technical tasks and Spike tickets with clear acceptance criteria, implementation detail and expected outcomesGood written communication is considered a core engineering skill.Technical OwnershipLead Engineers are expected to take ownership of services rather than simply deliver individual pieces of work.This includes:Owning services throughout their lifecycle, from design through to production supportUnderstanding the wider platform and the impact of technical decisionsDriving work forward with minimal supervisionIdentifying technical debt and proposing practical improvementsChallenging designs where there are operational, security, resilience or scalability concernsConsidering operational support as part of every solutionDesigning solutions that are maintainable by the wider engineering teamWorking across multiple repositories and servicesBeing accountable for the quality of technical deliverySupporting incident investigation and root cause analysis through to completionAgile Delivery & Ways of WorkingThe team follows Agile delivery practices and Lead Engineers are expected to actively contribute to both technical delivery and the running of the team.Expected experience includes:Working effectively within Agile delivery teamsParticipating in Sprint Planning, Sprint Reviews, Retrospectives and Daily Stand-upsLeading Agile ceremonies when required, with all engineers taking turns facilitating stand-ups, retrospectives and sprint planningBreaking down technical work into deliverable stories and tasksWriting high-quality Jira stories, technical tasks and Spike investigationsProducing clear acceptance criteria and technical implementation plansContributing to backlog refinement by helping define technical requirements and identify dependenciesProviding realistic estimates and identifying technical risks earlyCollaborating closely with Product Owners, Delivery Managers, Architects and other engineering teamsManaging priorities while balancing feature delivery, technical debt, operational support and platform improvementsCommunicating progress, blockers and technical decisions clearlyPromoting continuous improvement across engineering practices and deliveryLead Engineers are expected to help create an environment where knowledge is shared, engineering standards are maintained and the team continues to improve its ways of working.Leadership ExpectationsThis is a Lead DevOps Engineer position.Technical leadership is as important as technical delivery.Expected behaviours include:Supporting and mentoring permanent DWP engineers through coaching, pair programming and technical guidanceProviding technical direction within the squadLeading engineering discussions and helping the team reach sound technical decisionsMaking pragmatic technical decisions within agreed architectural standardsReviewing designs, code and merge requests, providing constructive feedbackIdentifying opportunities to improve existing processes, tooling and automation within approved technology boundariesMaintaining engineering standards across the teamProducing clear technical documentation to support delivery and future maintenanceCommunicating technical ideas, risks and recommendations to engineers, architects and stakeholdersWorking closely with Product Owners, Delivery Managers and Architects to shape technical deliveryInfluencing delivery through technical expertise rather than authorityOwning technical problems through to resolutionSetting a high engineering standard through the quality of your own workThe environment is governed by established technology standards, so improvements are generally expected within the approved tooling and architecture rather than introducing alternative products or platforms.Daily ActivitiesTypical work includes:Developing infrastructure using TerraformMaintaining reusable Terraform modulesDeveloping and maintaining Helm chartsSupporting and evolving reusable Helm chart librariesDeploying services to ECS and EKSSupporting the ECS platformWriting and maintaining GitLab CI pipelines and reusable pipeline templatesWriting and maintaining automated tests for infrastructure, platform code and supporting applicationsIntegrating automated testing into GitLab CI pipelinesInvestigating and resolving test failuresPerformance, load and stress testing using JMeter and k6Supporting production servicesInvestigating incidents and performing root cause analysisReviewing merge requestsParticipating in technical design discussionsWriting and maintaining technical documentationProducing Spike investigations and implementation proposalsWriting Jira stories, technical tasks and Spike ticketsImproving automation and engineering standardsWorking with application developers to support deploymentsMaintaining AWS platform componentsMentoring engineersLeading Agile ceremonies when requiredDriving continuous improvement across the platform and engineering practicesTechnologies UsedAWSECSEKSEC2TerraformTerraform TestsHelmGoKubernetesDockerGitLabGitLab CIKong API GatewayIstioOAuth 2.0OpenID Connect (OIDC)JWTIAMRBACCloudHSMAWS KMSSecrets ManagerSystems Manager Parameter StorePrometheusGrafanaDynatracePythonpytestNode.jsJMeterk6What Success Looks LikeAfter approximately three months in the role, a successful Lead DevOps Engineer would be expected to:Technical DeliveryBe comfortable deploying, supporting and troubleshooting services running on ECS and EKSDeliver production-ready Terraform, Helm and GitLab CI changes with minimal supervisionUnderstand the architecture of the Identity platform and how the various services interactBe comfortable working across multiple repositories, environments and AWS servicesSuccessfully troubleshoot production issues and participate in root cause analysisWrite well-tested code, infrastructure and automation with appropriate automated validationConfidently work with Kubernetes, Docker, Kong API Gateway, Istio and supporting AWS servicesProduce secure, maintainable and supportable engineering solutionsEngineering QualityProduce reusable, well-structured Terraform modules and Helm chartsContribute improvements to shared GitLab CI templates and reusable pipeline componentsMaintain high standards of testing across infrastructure, tooling and application codeIntegrate automated testing into CI/CD pipelines as part of normal deliveryWrite clear, maintainable and well-documented codeLeave services and repositories in a better state than they were foundDocumentationProduce high-quality Spike investigations with clear recommendations and outcomesWrite clear Jira stories, technical tasks and Spike tickets with meaningful acceptance criteriaMaintain accurate README files and implementation documentationProduce documentation that enables other engineers to understand, support and extend the platformClearly communicate technical decisions and the reasoning behind themLeadershipAct as a technical leader within the squadSupport and mentor less experienced engineers and permanent DWP staffLead engineering discussions and help drive technical decisionsProvide constructive feedback through code reviews and technical discussionsIdentify opportunities for improvement and help prioritise engineering effortTake ownership of services and technical problems through to completionBe recognised as someone other engineers naturally approach for technical guidanceAgile DeliveryActively contribute to Sprint Planning, Reviews, Retrospectives and Daily Stand-upsConfidently facilitate Agile ceremonies when requiredHelp refine and estimate upcoming workRaise technical risks and dependencies earlyBalance feature delivery with technical debt, operational support and platform improvementsWork collaboratively with Product Owners, Delivery Managers, Architects and other engineering teamsMindsetA successful Lead DevOps Engineer is someone who:Takes ownership rather than waiting to be directedLooks for ways to improve the platform, tooling and engineering practicesDelivers solutions that are maintainable, secure and operationally soundConsiders the long-term impact of technical decisionsShares knowledge freely and helps develop those around themLeads through technical expertise, collaboration and exampleContinuously learns and adapts to new technologies, engineering practices and platform requirementsThis job description is not to be regarded as exclusive or exhaustive. It is intended as an outline indicating of the areas of activity and may be amended from time to time in the light of the changing needs of the organisation.This role will require you to have or be willing to go through Security Clearance. As part of the onboarding process candidates will be asked to complete a Baseline Personnel Security Standard; details of the evidence required to apply may be found on the government website Gov.UK. If you are unable to meet this and any associated criteria, then your employment may be delayed, or rejected. Details of this will be discussed with you at interview. BenefitsBy joining us you can expect: Autonomy to develop and grow your skills and experience Be part of exciting project work that is making a difference in society Strong, inspiring and thought-provoking leadership A supportive and collaborative environment. As well as this, we offer: Development access to LinkedIn Learning, a management development programme and training Wellness 24/7 Confidential employee assistance programme Social – Breakfast Tuesdays, Thirsty Thursdays and Pizza on the last Thursday of each month as well as commitment to charitable causes Time off 25 days a year Pension Salary Exchange Scheme with 4% employer contribution and 5% employee contribution Discretionary Company Bonus based on company and individual performance Life Assurance of 4 times base salary Private Medical Insurance which is non-contributory (spouse and dependants included) Worldwide Travel Insurance which is non-contributory (spouse and dependants included) Job SummaryID: 65F93CDE36Department: Capabilities, IT Cloud