The Team

The AI Cluster Production Engineering team is part of the AI Compute Platform organization at Biohub, a non-profit research lab committed to open science and open-source AI. We own the design, operation, and reliability of large-scale multi-GPU AI clusters that power frontier AI biology research: protein language models, genomic foundation models, and scientific reasoning systems built to be shared, not monetized. Our clusters run Slurm on Kubernetes infrastructure and support everything from day-to-day AI researcher workflows to multi-node hero training runs at thousands of GPUs. The team works at the intersection of AI tooling, distributed systems, HPC, and frontier AI, debugging deep AI infrastructure problems and building AI systems critical to the entire AI organization.

The Opportunity

CZ Biohub's mission is to cure or prevent all human disease. Achieving that requires training frontier-scale AI biology models, and that demands reliable, high-performance compute infrastructure. This is production engineering work at a frontier AI lab, with the twist that the mission is biology and the science is open. You'll keep GPU clusters running at high utilization, debug the toughest distributed systems failures, and build the operational foundations for scaling to multi-thousand GPU hero runs.  The technical problems are genuinely hard (e.g., multi-node distributed training, InfiniBand fabrics, large-scale storage, Slurm at scale) inside an organization where the work is aimed at helping people, not optimizing ad revenue.

What You'll Do

• Own reliability, observability, and incident response for multi-site GPU clusters running Slurm on Kubernetes. Build the systems, automation, and processes that keep clusters healthy,  and that enable fast, efficient recovery when things break.
• Debug and resolve deep infrastructure failures across storage, networking, scheduling, and GPU compute layers. Build the tooling and operational patterns that make these failures easier to detect, diagnose, and prevent.
• Design and execute GPU cluster scaling plans, systematically validating storage, networking, interconnect, and scheduler behavior as clusters grow to support larger training runs.
• Build automation and tooling to manage cluster operations at scale: capacity planning, GPU utilization monitoring workload manager policy management, and pod lifecycle automation.
• Drive configuration-as-code practices, ensuring cluster state is reproducible and auditable, and managed through version-controlled pipelines.
• Collaborate directly with AI researchers and hero run leads to understand training workload patterns and design infrastructure that meets frontier-scale requirements.
• Own the vendor relationship on technical issues — escalating SEV1s, coordinating across multiple partners and network backbone teams, holding them accountable to root/proximate cause analysis and SLAs.
• Contribute to capacity planning: projecting GPU demand, managing cluster expansion across GPU generations, and coordinating multi-cluster strategy.
• Improve operational resilience, reducing mean time to detect and resolve incidents, reducing toil through automation, and developing runbooks that scale the team's operational knowledge beyond any individual.

What You'll Bring

• 8+ years of AI/ML infrastructure engineering experience, with deep expertise in at least one of: HPC/Slurm cluster operations, Kubernetes at scale, distributed systems debugging, or GPU compute infrastructure.
• Strong Linux systems fundamentals — networking (TCP/IP, InfiniBand, RDMA, MTU/MSS/PMTUD), storage (NFS, VAST, WEKA, POSIX semantics), kernel internals (cgroups, namespaces, eBPF, sysctls).
• Hands-on experience with Kubernetes and cloud-native infrastructure — pod lifecycle, CNI plugins (Cilium preferred), StatefulSets, Helm, ArgoCD, or equivalent GitOps tooling.
• Experience with HPC workload managers — Slurm strongly preferred (QoS, partitions, preemption, accounting, Sunk/CoreWeave patterns a plus).
• Debugging instinct: ability to form hypotheses quickly, design controlled experiments, and root cause complex multi-system failures under pressure. You enjoy finding the hard bugs.
• Proficiency in Python and Bash for automation and tooling. Go, Rust, or C/C++ a plus.
• Experience with observability stacks — Prometheus/VictoriaMetrics, Grafana, DCGM metrics, distributed tracing. You know how to instrument systems you don't control.
• Excellent communication — you can write a crisp incident summary for researchers, a technical escalation to a vendor CTO, and a system design doc for teammates, all in the same day.
• Bonus: experience with distributed AI training infrastructure (NCCL, PyTorch DDP, multi-node job debugging, checkpoint/restart patterns, container environments for large-scale training).

Compensation

The Redwood City, CA base pay range for a new hire in this role is $241,000 - $331,000. New hires are typically hired into the lower portion of the range, enabling employee growth in the range over time. Actual placement in range is based on job-related skills and experience, as evaluated throughout the interview process. 

Better Together

As we grow, we’re excited to strengthen in-person connections and cultivate a collaborative, team-oriented environment. This role is a hybrid position requiring you to be onsite for at least 60% of the working month, approximately 3 days a week, with specific in-office days determined by the team’s manager. The exact schedule will be at the hiring manager's discretion and communicated during the interview process.

Benefits for the Whole You 

We’re thankful to have an incredible team behind our work. To honor their commitment, we offer a wide range of benefits to support the people who make all we do possible. 

• Provides a generous employer match on employee 401(k) contributions to support planning for the future.
• Paid time off to volunteer at an organization of your choice. 
• Funding for select family-forming benefits. 
• Relocation support for employees who need assistance moving

If you’re interested in a role but your previous experience doesn’t perfectly align with each qualification in the job description, we still encourage you to apply as you may be the perfect fit for this or another role.

#LI-Hybrid 

The Team

Our AI research team sits at the heart of our mission to unlock new dimensions of biological understanding. You will leverage state-of-the-art AI to accelerate discovery and drive transformative insights in biology—developing novel AI models purpose-built for biological research, engineering robust systems that enable breakthrough science at unprecedented scale, and translating these advances into practical tools that empower researchers worldwide.

Our approach is comprehensive and integrated, bringing together world-class AI model development, exceptional engineering talent, high-quality biological data, powerful computing infrastructure, and strategic partnerships. Success requires excellence across five interconnected pillars: training frontier AI models specifically for biology; building engineering systems that maximize research velocity and efficiency; executing a sophisticated data strategy that fuels AI development; operating a world-class AI compute platform; and creating impactful products that transform AI capabilities into accessible scientific tools.

The Opportunity

The Data Engineering and Infrastructure team brings AI/ML technology and Data to the table in new ways to help drive AI powered solutions that accelerate Biomedical research. We are uniquely positioned to design, build, and scale software systems to help scientists better address the myriad challenges they face. We are supporting researchers and scientists around the world by developing the capacity to apply state-of-the-art methods in artificial intelligence and machine learning to solve important problems in the biomedical sciences.

This team works on building shared tools and platforms to be used across Biohub, partnering and supporting the work of an extensive group of Research Scientists, Data Scientists, AI Research Scientists, and Computational Biologists. Members of the shared infrastructure engineering team have an impact on all of Biohub’s initiatives by enabling the technology solutions used by other engineering teams to build a frontier model and scale the feedback loop.

What You'll Do

• Develop and maintain the tooling and infrastructure that drives the entire data lifecycle at Biohub, from ingestion and processing to secure storage and access. Your work will directly support cutting-edge research, AI model training, and data analysis across the organization.
• Partner with researchers and engineers across various domains, including genetics, imaging, and scientific literature. You'll work on various use cases, from web analytics to complex model training, ensuring data accessibility and performance for teams across CZI.
• Design and implement flexible, scalable, and performant systems to address our stakeholders’ needs, leveraging technologies like Argo Workflows, Slurm, Ray, AWS Parallel Cluster for mass-scale job processing and orchestration; Vast Data, Delta Lake, Databricks and Apache Iceberg for data management and access; and cloud and on-prem HPC resources

What You'll Bring

• Education & Experience: BS, MS, or PhD in Computer Science or a related technical discipline, or equivalent experience. 8+ years of hands-on coding experience in scripting (Python, PHP, Ruby) and systems languages (Rust, C++, C#, Go, Java, or Scala).
• Data Platform Expertise: Proficiency in managing large-scale data operations, including designing scalable pipelines (streaming and batch), working with varied data types, and optimizing flexible storage solutions using tools like Argo Workflows, Databricks, Slurm, etc.
• Data Management and Development Operations: Experience with data governance, metadata, and data lineage tooling like Open Lineage or Marquez. Deep experience working with building CI/CD pipelines for data infrastructure and associated observability and monitoring tooling such as Prometheus, Grafana, OpenTelemetry, Prometheus, or Honeycomb
• Supporting Complex Data and AI ML Workflows: Experience with addressing end to end data needs for working with complex data and delivering this data ready form model training, working directly with AI Researchers and AI Engineers as part of AI model training project teams. Knowledge of data pipelines that feed Deep Learning model training that makes use of accelerated computing systems (GPUs as well as ASICs, FPGAs, TPUs)
• Containerization: Extensive experience with scaling containerized applications on Kubernetes or Mesos, with a focus on secure custom containers, replicability and portability.
• Cloud & Infrastructure: Strong experience with AWS, GCP, or Azure. Exposure to hybrid environments with on-prem and colocation systems is a plus. Familiarity with Infrastructure as Code (e.g., Terraform, Ansible) and monitoring tools (Datadog, Prometheus).
• Collaboration & Problem Solving: Proven ability to work with diverse, cross-functional stakeholders and teams to navigate complex technical challenges, adapt to evolving requirements, and drive impactful solutions.

Compensation

The Redwood City, CA base pay range for a new hire in this role is $214,000 - $295,000. New hires are typically hired into the lower portion of the range, enabling employee growth in the range over time. Actual placement in range is based on job-related skills and experience, as evaluated throughout the interview process. 

This position may be eligible to participate in our discretionary annual performance bonus program. Bonus eligibility and targets are determined in accordance with our total rewards philosophy and may vary by role.

Better Together

As we grow, we’re excited to strengthen in-person connections and cultivate a collaborative, team-oriented environment. This role is a hybrid position requiring you to be onsite for at least 60% of the working month, approximately 3 days a week, with specific in-office days determined by the team’s manager. The exact schedule will be at the hiring manager's discretion and communicated during the interview process.

Benefits for the Whole You 

We’re thankful to have an incredible team behind our work. To honor their commitment, we offer a wide range of benefits to support the people who make all we do possible. 

• Provides a generous employer match on employee 401(k) contributions to support planning for the future.
• Paid time off to volunteer at an organization of your choice. 
• Funding for select family-forming benefits. 
• Relocation support for employees who need assistance moving

If you’re interested in a role but your previous experience doesn’t perfectly align with each qualification in the job description, we still encourage you to apply as you may be the perfect fit for this or another role.

#LI-Hybrid