AI Infrastructure Engineer (Member of Technical Staff)

• We are looking for an AI Infra engineer to join our growing team. We work with Kubernetes, Slurm, Python, C++, PyTorch, and primarily on AWS. As an AI Infrastructure Engineer, you will be partnering closely with our Inference and Research teams to build, deploy, and optimize our large-scale AI training and inference clusters
• Design, deploy, and maintain scalable Kubernetes clusters for AI model inference and training workloads
• Manage and optimize Slurm-based HPC environments for distributed training of large language models
• Develop robust APIs and orchestration systems for both training pipelines and inference services
• Implement resource scheduling and job management systems across heterogeneous compute environments
• Benchmark system performance, diagnose bottlenecks, and implement improvements across both training and inference infrastructure
• Build monitoring, alerting, and observability solutions tailored to ML workloads running on Kubernetes and Slurm
• Respond swiftly to system outages and collaborate across teams to maintain high uptime for critical training runs and inference services
• Optimize cluster utilization and implement autoscaling strategies for dynamic workload demands

Strong expertise in Kubernetes administration, including custom resource definitions, operators, and cluster managementHigh level familiarity with LLM architecture and training processes (Multi-Head Attention, Multi/Grouped-Query, distributed training strategies)Experience with deploying and managing distributed training systems at scaleDeep understanding of container orchestration and distributed systems architectureExperience managing GPU clusters and optimizing compute resource utilizationHands-on experience with Slurm workload management, including job scheduling, resource allocation, and cluster optimizationExpert-level Kubernetes administration and YAML configuration managementPython and C++ programming with focus on systems and infrastructure automationProficiency with Slurm job scheduling, resource management, and cluster configurationExperience developing APIs and managing distributed systems for both batch and real-time workloadsStrong understanding of networking, storage, and compute resource management for ML workloadsHands-on experience with ML frameworks such as PyTorch in distributed training contextsSolid debugging and monitoring skills with expertise in observability tools for containerized environmentsExperience with Kubernetes operators and custom controllers for ML workloadsAdvanced Slurm administration including multi-cluster federation and advanced scheduling policiesFamiliarity with GPU cluster management and CUDA optimizationExperience with other ML frameworks like TensorFlow or distributed training librariesBackground in HPC environments, parallel computing, and high-performance networkingKnowledge of infrastructure as code (Terraform, Ansible) and GitOps practicesExperience with container registries, image optimization, and multi-stage builds for ML workloadsIdeally, 3-5 years of relevant experience in ML systems deployment with specific focus on cluster orchestration and resource managementExperience supporting both long-running training jobs and high-availability inference servicesProven track record with Slurm cluster administration and HPC workload managementDemonstrated experience managing large-scale Kubernetes deployments in production environmentsPrevious roles in SRE, DevOps, or Platform Engineering with focus on ML infrastructure