Senior Principal Cloud Development Engineer in City of Westminster

The Senior Principal, Operations Resiliency & Game Day Strategy within Cloud Operations is responsible for defining and operating the enterprise Game Day program that validates the resiliency of cloud‑hosted applications and platforms.

You will design failure scenarios drawn from real production incidents, select and prioritize applications for testing, and ensure that exercise outcomes translate directly into improved runbooks, recovery paths, and architectural resilience.

Responsibilities

• Define and operationalize the enterprise Game Day strategy that validates resiliency of cloud‑hosted applications and platforms.
• Establish application selection criteria, testing frequency, and a scalable operating model for controlled failure testing.
• Develop a taxonomy of application patterns and map them to targeted, reusable fault injection scenarios drawn from real production failures.
• Oversee Game Day execution, both manual and automated, validating observability, detection, response, and recovery workflows.
• Close the loop between production incidents and simulated testing: incident, scenario, test, finding, mitigation, and retest.
• Drive identification of missing recovery paths, observability gaps, and ineffective automation, then ensure findings are tracked and assigned to accountable owners.
• Push execution from manual to automated: fault injection, scenario orchestration, metrics capture, and reporting.
• Define and track effectiveness metrics such as time to detect, time to recover, automated recovery success rates, and reduction in repeat incident patterns.
• Partner across Incident Management, Application Teams, Platform Engineering, and SRE to ensure Game Day outputs translate directly into improved runbooks, recovery paths, and architectural resilience.

Qualifications

• Minimum of 10 years of related experience.
• Bachelor's degree preferred or equivalent experience.
• Deep experience in cloud operations, SRE, or resiliency engineering.
• Hands‑on knowledge of incident management and root‑cause analysis.
• Experience with fault injection tools such as AWS FIS, Gremlin, or equivalent.
• Experience with AI‑assisted operations tooling such as AWS Dev Ops Agent or equivalent.
• Familiarity with policy‑as‑code frameworks such as OPA, Sentinel, or equivalent.
• Strong understanding of distributed‑systems failure modes, observability architectures, and automation/runbook engineering.
• Ability to operate horizontally across engineering and operations organizations – influencing without direct delivery ownership.
• Strong judgment on what to test, when, and why.

Benefits

Competitive compensation, including base pay and annual incentive; comprehensive health and life insurance and well‑being benefits; pension; paid time off and personal/family care and other leaves of absence.

#J-18808-Ljbffr