High Performance Computing System Architect - Scientific Computing & Data

at  Mount Sinai

QUALIFICATIONS

• Bachelor’s degree in computer science, engineering or another scientific field. Master’s or PhD preferred
• 6 years of progressive HPC system administration and operations (preferably in a Redhat/CentOS Linux administration, Batch HPC cluster environment)
• Must be an expert troubleshooter; Must be a team player and customer focused
• Experience with configuration management systems such as xCAT, Puppet and/or Ansible
• Experience with networking and security
• Experience with Infiniband and Gigabit Ethernet
• Experience with LSF and GPFS Spectrum Scale parallel file systems and storage
• Excellent communication skills, analytical ability, strong judgment and management skills, and the ability to work effectively as a liaison between both research and technology teams.
• Strong written, oral, and interpersonal communication skills
• Script and programming experience
• Ability to gain buy-in from stakeholders to resolve significant architecture issues
• Ability to manage multiple priorities, commitments and projects.
• Ability to lead the project to successful completion with little guidance

PREFERRED EXPERIENCE:

• Experience with archival storage and tape libraries (TSM) is highly preferred
• Experience with databases and web services is highly preferred
• Experience with supporting Direct Liquid Cooling equipment
• Compliance, HIPAA
• Experience with managing web access to HPC resources (such as Open OnDemand)
• Singularity and/or docker containers
• Academic and/or healthcare research setting
• Nagios
  Employer Description

The Scientific Computing and Data group at the Icahn School of Medicine at Mount Sinai partners with scientists to accelerate scientific discovery. To achieve these aims, we support a cutting-edge high-performance computing and data ecosystem along with MD/PhD-level support for researchers. The group is composed of a high-performance computing team, the research clinical data warehouse team and a research data services team.
The HPC Architect, as a member of the Scientific Computing and Data group, is responsible for the architecture, design and deployment of Scientific Computing’s computational and data science ecosystem. This ecosystem includes high-performance computing (HPC) systems, clinical research databases, and a software development infrastructure for local and national projects. To meet Sinai’s scientific and clinical goals, the Architect has a deep technical understanding of the best practices for computational, data and software development systems along with a strong focus on customer service for researchers. The HPC Architect is an expert troubleshooter, productive team member, and can lead the project to successful completion with little guidance. The incumbent is a productive partner for researchers and technologists throughout the organization and beyond. This position reports to the Director for Computational & Data Ecosystem in Scientific Computing & Data. Specific responsibilities are outlined below.

Responsibilities

• Responsible for the technical operations including the architect, design, expansion, monitoring, support, and maintenance for Scientific Computing’s computational and data science ecosystem consistent with best practices. Key components include ~30,000 cores, with high bandwidth, low latency interconnects, GPUs, large shared memory nodes, databases, web servers, scientific workflows and 40+ petabytes of storage in production, clinical data warehouse and software development environment.
• Maintains, tunes and manages computational, data, cloud technologies and workflow systems for MSSM researchers, scientists and their external collaborators. Defines and deploys a comprehensive computational and data vision. Identifies and communicates system advantages/disadvantages and tradeoffs.
• Troubleshoots isolates and resolves application, system and other technical problems (hardware, software and network). Actively monitors the systems.
• Design, develop, implement all system administration tasks, including hardware and software configuration, configuration management, system monitoring (including the development and maintenance of regression tests), usage reporting, system performance (file systems, scheduler, interconnect, high availability, etc.), security, networking and metrics, etc.
• Researches, deploys and optimizes resource management and scheduling software and policies and actively monitoring.
• Designs, tunes, manages and upgrades parallel file systems, storage and data-oriented resources.
• Participates in the integration of HPC resources with laboratory equipment such as sequencers, clinical and research data resources and systems, etc. Incorporate and link data and compute resources.
• Develops innovations with researchers for their projects, designs and implements frameworks, pipelines and infrastructure interface for enhanced operations and performance; Ensures the technical design and operation of the HPC ecosystem is efficient and productive for research.
• Researches, deploys and manages security infrastructure, including development of policies and procedures. Collaborates effectively with research and hospital system IT, compliance, HIPAA, security and other departments to ensure compliance with all regulations and Sinai policies.
• Partners with other peers regionally, nationally and internationally to discover, propose and deploy a world-class research infrastructure for Mount Sinai.
• Assists in developing and writing system design for research proposals.
• Works effectively and productively with other team members within the group and across Mount Sinai. Works as a strong team player. Provide after hours support in case of a critical system issue.
• Performs related duties as assigned or requested.