Principal Software Reliability Engineer, Product Engineering

Join us at Entrust. We’re shaping the future of identity-centric security solutions. From our comprehensive portfolio of solutions to our flexible, global workplace, we empower careers, foster collaboration, and build solutions that help keep the world moving safely.

Headquartered in Minnesota, Entrust is an industry leader in identity-centric security solutions, serving over 150 countries with cutting-edge, scalable technologies. But our secret weapon? Our people. It’s the curiosity, dedication, and innovation that drive our success and help us anticipate the future.

This is a Product Reliability position, not an infrastructure SRE role. Our DevOps team manages the infrastructure platform; this role focuses on application and service-level reliability, working directly with product engineers. This is the first role of its kind in product engineering. Reporting to the VP of Product Engineering for Consumer Identity, you’ll drive reliability efforts across the team: defining the roadmap, prioritising initiatives, and partnering with engineering directors and senior ICs to deliver them.

Why Join Us

• Greenfield opportunity: You’ll define Product Reliability as a discipline here. Build the playbook, not inherit one.
• High-impact domain: Consumer Identity powers identity verification and biometric authentication for some of the world’s largest financial institutions. Our reliability directly impacts fraud prevention and customer onboarding at scale.
• Real authority: Direct line to VP Engineering, budget for tooling, seat at architecture council and service reviews.
• Strong foundation: We’re not firefighting. 99.98% uptime means you’re optimising, not triaging chaos.
• Technical depth: Work across ML pipelines, computer vision systems, and mobile SDKs (not just YAML and dashboards).
• Ownership culture: Engineers own their services end-to-end; you’ll amplify that, not replace it.

Experience Level

• Staff SRE: 8+ years in software engineering, 4+ years in reliability/SRE. Drives reliability initiatives across multiple teams; hands-on with complex systems.
• Principal SRE: 15+ years in software engineering, 6+ years in reliability/SRE. Sets technical direction org-wide; influences business-unit-level reliability strategy.

We’re open to either level. Scope and compensation will match your experience. Principal candidates should demonstrate cross-org impact and a track record of building reliability programs from scratch.

Current State Incident Analysis (2020–2025)

• Postmortem volume peaked in 2023, down 48% since then despite increased release cadence.
• P0-to-P1+ ratio remains stable despite lower overall incident volume.
• 65% change-induced incidents (deployments, migrations, config changes); 35% organic (third-party outages, expirations, attacks).
• Change-induced ratio improved modestly: 69% → 62%.
• Detection time: 35 min → 18 min.
• Customer-first detection: 40% → 22%.

Availability Targets

• 2024 & 2025 average uptime: 99.98% (as available in our public status page).
• Goal: Consistent 99.99% (four nines) average uptime, SLO breach reductions.

System Simplification

• We’re reducing system complexity to narrow the reliability target area: Microservices (K8s deployments/rollouts) reduced 29% from peak, with further cuts planned for 2026.
• Goal: Smaller footprint, higher reliability, lower cost for new regions.

Role Objectives

• Primary goal: Improve release safety, reduce releases that cause downtime or SLO degradation. We already have foundational systems in place:

• Automated test coverage and crowd testing.
• A/B testing and dark canaries.
• Progressive rollouts (infrastructure and application level).
• Back-testing against historical data.

Ideal Candidate Profile Mindset

• Passionate about reliability as a discipline, not just a checkbox.
• Focused on reliability, not product features, but willing to learn the product to understand impact.
• Hands-on: eager to build tooling and systems.
• Pragmatic about balancing reliability with development velocity.

Required Skills

• Software Engineering: Strong software engineering in at least one of our backend languages (Python, Ruby, Node.js); able to navigate most of our codebase. Experience building reliability tooling: progressive delivery, automated rollbacks, monitoring/alerting.
• Reliability Patterns: Deep knowledge of resilience patterns: circuit breakers, bulkheads, back-pressure, retries with backoff, rate limiting, load shedding, graceful degradation. Solid incident management and blameless postmortem practices.
• Observability: Proficiency with observability: distributed tracing, structured logging, metrics instrumentation. Uses data to drive decisions: experienced with SLIs, SLOs, and error budgets.
• Communication: Skilled at influencing without authority. Able to hold deep technical reliability discussions with senior ICs.

Nice-to-Have

• Experience with chaos engineering (fault injection, game days, controlled failure experiments).
• ML system reliability experience (mixed I/O and CPU-bound workloads, non-deterministic behaviour, model serving).
• Familiarity with our specific stack (Datadog, Kubernetes, AWS, GitLab CI/CD).
• Experience leveraging LLMs for code analysis, design doc review, or automated runbook generation.
• On-call experience in a high-availability environment.

Our Stack

• Backend: Python, Ruby on Rails, Node.js.
• Frontend: React, TypeScript.
• Mobile: Swift (iOS), Kotlin (Android), React Native.
• Infrastructure: AWS, Kubernetes, Terraform, SNS, SQS.
• Databases: PostgreSQL (Aurora), Redis, OpenSearch.
• Observability: Datadog, Splunk, Sentry.
• ML: PyTorch, TensorFlow.
• CI/CD: GitLab (on-prem).

At Entrust, we don’t just offer jobs – we offer career journeys. Here is what you can expect when you join our team:

• Career Growth: Whether you’re a budding developer or a seasoned expert, we’re invested in your professional journey. With learning-forward initiatives and exciting challenges, your growth is our priority.
• Flexibility: Life is all about balance. Whether you’re remote, hybrid, or on-site, we offer flexible options that fit your lifestyle.
• Collaboration: Here, your voice matters. Our teams thrive on sharing ideas, brainstorming solutions, and working together to build a better tomorrow.

We believe in securing identities—but it doesn’t stop there. At Entrust, we’re passionate about valuing all identities. Our culture is built on diversity, inclusion, and respect. From unconscious bias training for our leaders to global affinity groups that connect colleagues across the globe, we’re creating a community where everyone is encouraged to be themselves.

Ready to Make an Impact? If you’re excited by the prospect of innovating, growing your career, and collaborating in a dynamic environment, Entrust is the place for you. Join us in making a difference. Let’s build a more secure world—together. Apply today!