Engineering Manager, Hardware Health

at  OpenAI

ABOUT THE TEAM

OpenAI’s Hardware Health team is dedicated to ensuring the optimal performance and reliability of our custom-built hyperscale supercomputers. We focus on maximizing supercomputing capacity for research and ensuring that our researchers are minimally impacted by hardware faults. This team is critical in maintaining the infrastructure that supports cutting-edge AI research at OpenAI.
The Hardware Health team operates within the broader Platform organization, which is incubated inside OpenAI’s Research team. Our work is on the front lines of innovation, supporting the engineering and research required to train large-scale AI models of unprecedented capability.

ABOUT THE ROLE

As an Engineering Manager on the Hardware Health team, you will lead a group of engineers responsible for ensuring the reliability, performance, and scalability of our supercomputing infrastructure. Your team will build infrastructure and automation to manage the lifecycle and operations of our cutting-edge hyperscale supercomputers. This includes monitoring, detecting, diagnosing, and resolving issues, as well as operationalizing scalable solutions.
Our team heavily uses Python and some Rust, and we are looking for someone who is excited to work in this stack and learn it if they’re not familiar.
This role is based in San Francisco, CA. We use a hybrid work model of 3 days in the office per week and offer relocation assistance to new employees.

IN THIS ROLE, YOU WILL:

• Lead a team of software engineers to manage and optimize the critical infrastructure of our hyperscale supercomputers.
• Develop and implement strategies to monitor and maintain system health, ensuring minimal downtime and optimal performance.
• Collaborate closely with hardware engineers, systems engineers, and researchers to understand the infrastructure requirements and challenges of our AI training workloads.
• Drive the development of tools and automation to detect, diagnose, and mitigate hardware health-related issues.
• Build a team culture that prioritizes reliability, scalability, and performance, while fostering innovation and continuous learning.
• Create a diverse, equitable, and inclusive environment that encourages team members to contribute their best ideas and challenge assumptions.

YOU MIGHT THRIVE IN THIS ROLE IF YOU:

• Have 5+ years of experience in engineering management, with a focus on large-scale infrastructure roles.
• Possess deep expertise in managing and optimizing hardware and software in large-scale, high-performance computing environments.
• Are a seasoned technical leader, excited to do hand-on technical work but equally excited to lead technical teams to peak performance.
• Have a strong background in troubleshooting complex system issues and developing solutions to prevent future occurrences.
• Bonus: Familiarity with modern AI infrastructure technologies (e.g., A100s, H100s, etc.), or experience working in large-scale HPC environments is a plus.
• Are skilled in collaborating across teams to solve complex problems and deliver high-impact results.
• Care deeply about diversity, equity, and inclusion, and have a track record of building inclusive teams.
• Have excellent communication skills, with the ability to convey complex technical concepts to both technical and non-technical audiences.
• Have a humble attitude, an eagerness to help your colleagues, and a desire to do whatever it takes to make the team succeed.
• Own problems end-to-end, and are willing to pick up whatever knowledge you’re missing to get the job done.