Senior Production Engineer - Senior Vice President

The Production Engineer is a pivotal role within Citi's Technology organisation, responsible for designing, building, and operating the intelligent systems that underpin our global production environment.

This is an engineering-first position at the intersection of software craftsmanship, AI-native development, and large-scale distributed systems.

As part of a multi-year transformation journey, the successful candidate will help define what production engineering looks like in an era of autonomous agents, generative AI, and self-healing infrastructure.

You will be expected to write production‑grade code daily, design agentic workflows, and contribute meaningfully to the evolution of our AI engineering practices across Citi's technology hub.

The role requires a comprehensive understanding of multiple areas within a function and how they interact to achieve the objectives of the function.

It requires an in-depth understanding of the business impact of technical contributions and is accountable for delivery of a full range of end‑to‑end projects.

Excellent communication skills are required to negotiate internally, and involvement in short‑to‑medium term planning of actions and resources for your own area is expected.

Responsibilities

• Design, develop, and maintain production‑grade software systems with a strong emphasis on reliability, scalability, and operational excellence across Citi's global technology estate.
• Architect and implement agentic AI workflows—building autonomous systems that can reason, plan, and act across production environments with minimal human intervention.
• Apply advanced prompt engineering techniques to integrate large language models (LLMs) into operational tooling, incident response pipelines, and developer productivity platforms.
• Lead the development of AI‑native observability solutions—leveraging intelligent agents to detect anomalies, predict failures, and automate remediation before issues impact end users.
• Write clean, well‑tested, and well‑documented code across the full stack; champion engineering best practices including code review, pair programming, and test‑driven development.
• Drive continuous delivery and automation efforts across supported applications by means of root‑cause analysis reviews, knowledge management, performance tuning, and user training.
• Operate and evolve CI/CD pipelines, Infrastructure‑as‑Code tooling, and Git Ops workflows to support rapid, safe delivery of software at scale.
• Collaborate with platform, data, and product engineering teams to embed AI capabilities into the production lifecycle—from deployment to decommission.
• Implement the Agile framework through one of its implementations (SCRUM or Kanban) and ensure it integrates with overall organisational processes.
• Operate within a highly regulated financial environment, maintaining an in‑depth understanding of compliance requirements and their implications for system design and data handling.
• Coach and mentor team members on AI engineering practices, prompt design patterns, and agentic system architecture, fostering a culture of continuous learning and technical excellence.
• Communicate progress and project status across the organisation, ensuring stakeholders are managed appropriately throughout the execution period.
• Foster a culture that promotes transparency and innovation to increase team productivity.

Qualifications

• Demonstrable experience in a critical software engineering or production engineering role with high business impact and a strong programming foundation (Java, Python, Go, or equivalent).
• Hands‑on experience with AI/ML engineering—including working with LLM APIs (Open AI, Anthropic, Gemini, or open‑source equivalents), embedding models, and vector databases.
• Proven expertise in prompt engineering: designing, iterating, and evaluating prompts for production use cases including classification, summarisation, code generation, and autonomous decision‑making.
• Experience designing and deploying agentic systems using frameworks such as Lang Chain, Lang Graph, Auto Gen, Crew AI, or equivalent—including multi‑agent orchestration and tool‑use patterns.
• Excellent engineering skills and strong understanding of Software Development Lifecycle, Git Ops, and modern Dev Sec Ops practices.
• Excellent working knowledge of key computer science concepts (networking, operating systems, virtualisation, containerisation, etc.).
• Polyglot full‑stack developer mentality and ability to pick up new languages and skills.
• Excellent debugging and analytical skills: ability to isolate root cause across networking/infrastructure, application, and database stacks.
• Operational experience deploying and running services at scale on top of Docker/Kubernetes stack and a service mesh (Istio or equivalent) highly desirable.
• Experience with orchestration tools for CI/CD and Infrastructure‑as‑Code tooling (Terraform, Cloud Formation, Pulumi, etc.) highly desirable.
• Experience delivering software using Agile delivery methodologies (SCRUM/Kanban) mandatory.
• Operational experience using middleware technologies (MQ, Apache Kafka, etc.) to run services at scale desirable.
• Strong experience with end‑to‑end observability stacks (Datadog, App Dynamics, Dynatrace, etc.) desirable.
• Degree in Computer Science, Mathematics, Physics, or a related technical subject desirable.
• Experience managing senior stakeholders.
• Consistently demonstrates clear and concise written and verbal communication skills.
• Ability to operate in a global environment with on‑, near‑, or off‑shore matrix reporting structures.
• Human Qualities & Soft Skills
• Learnability — rapidly acquires new skills, frameworks, and paradigms.
• Teachability — receives feedback with openness and intellectual humility, seeks mentorship, and applies guidance.
• Flexibility & Adaptability — thrives in ambiguity, pivots gracefully when requirements shift or priorities change.
• Engineering Mindset — approaches every problem systematically: decomposing complexity, forming hypotheses, and validating solutions with rigor and precision.
• Product‑Minded Thinking — understands that code serves users and business outcomes, balancing technical elegance with pragmatic delivery.
• Collaborative Spirit — builds trust across disciplines, elevating the team's collective output through generosity and clear communication.
• Intellectual Curiosity — asks “why” before “how”, exploring the edges of what's possible with AI and production systems.
• Ownership & Accountability — takes end‑to‑end responsibility for what they build, following through from design to deployment to post‑incident review.
• What We’ll Provide You
• 27 days annual leave (plus bank holidays)
• Discretionary annual performance‑related bonus
• Private medical care & life insurance
• Employee assistance program
• Pension plan
• Paid parental leave
• Special discounts for employees, family, and friends
• Access to learning and development resources

Alongside these benefits Citi is committed to ensuring our workplace is where everyone feels comfortable coming to work as their whole self, every day.

• #LI-BH1
• Equal Opportunity Statement

Citi is an equal opportunity employer, and qualified candidates will receive consideration without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other characteristic protected by law.

If you are a person with a disability and need a reasonable accommodation to use our search tools and/or apply for a career opportunity review accessibility at Citi.

View Citi’s EEO policy statement and the Know Your Rights poster.

#J-18808-Ljbffr