Scale Labs, Research Scientist — Agent Robustness

As the leading data and evaluation partner for frontier AI companies, Scale plays an integral role in understanding the capabilities and safeguarding AI models and systems. Building on this expertise, Scale Labs has launched a new team focused on policy research, to bridge the gap between AI research and global policymakers to make informed, scientific decisions about AI risks and capabilities.

Our research tackles the hardest problems in agent robustness, AI control protocols, and AI risk evaluations to help governments, industry, and the public understand and mitigate AI risk while maximizing AI adoption. This team collaborates broadly across industry, the public sector, and academia and regularly publishes our findings. We are actively seeking talented researchers to join us in shaping this vision.

As a Research Scientist working on Agent Robustness you will work on the fundamental challenges of building AI agents that are safe and aligned with humans. For example, you might: 

• Research the science of AI agent capabilities with a focus on how they relate to safety, risk factors, and methodologies for benchmarking them;
• Design and build harnesses to test AI agents’ tendency to take harmful actions when pressured to do so by users or tricked into doing so by elements of their environment;
• Design and build exploits and mitigations for new and unique failure modes that arise as AI agents gain affordances like coding, web browsing, and computer use;
• Characterize and design mitigations for potential failure modes or broader risks of systems involving multiple interacting AI agents.

Ideally you’d have:

• Commitment to our mission of promoting safe, secure, and trustworthy AI deployments in the industry as frontier AI capabilities continue to advance.
• Practical experience conducting technical research collaboratively. You should be comfortable building and leveraging agent scaffolding, designing evaluation harnesses, and quickly turning new ideas from the research literature into working prototypes.
• Experience with post-training and RL techniques such as RLHF, DPO, GRPO, and similar approaches.
• A track record of published research in machine learning, particularly in generative AI.
• At least three years of experience addressing sophisticated ML problems, whether in a research setting or in product development.
• Strong written and verbal communication skills to operate in a cross-functional team.

Nice to have:

• Hands-on experience with agent evaluation frameworks such as SWE-bench, WebArena, OSWorld, Inspect, or similar tools.
• Experience with red-teaming, prompt injection, or adversarial testing of AI systems.

Our research interviews are crafted to assess candidates' skills in practical ML prototyping and debugging, their grasp of research concepts, and their alignment with our organizational culture. We will not ask any LeetCode-style questions. If you’re excited about advancing AI safety and contributing to our mission, we encourage you to apply, even if your experience doesn’t perfectly align with every requirement.

Scale works with the industry’s leading AI labs to provide high quality data and accelerate progress in GenAI research. We are looking for Research Scientists and Research Engineers with expertise in LLM post-training (SFT, RLHF, reward modeling). This role will focus on optimizing data curation and eval to enhance LLM capabilities in both text and multimodal modalities.

In this role, you will develop novel methods to improve the alignment and generalization of large-scale generative models. You will collaborate with researchers and engineers to define best practices in data-driven AI development. You will also partner with top foundation model labs to provide both technical and strategic input on the development of the next generation of generative AI models.

You will:

• Research and develop novel post-training techniques, including SFT, RLHF, and reward modeling, to enhance LLM core capabilities in both text and multimodal modalities.
• Design and experiment new approaches to preference optimization.
• Analyze model behavior, identify weaknesses, and propose solutions for bias mitigation and model robustness.
• Publish research findings in top-tier AI conferences.

Ideally you’d have:

• Ph.D. or Master's degree in Computer Science, Machine Learning, AI, or a related field.
• Deep understanding of deep learning, reinforcement learning, and large-scale model fine-tuning.
• Experience with post-training techniques such as RLHF, preference modeling, or instruction tuning.
• Excellent written and verbal communication skills

• Published research in areas of machine learning at major conferences (NeurIPS, ICML, ICLR, ACL, EMNLP, CVPR, etc.) and/or journals
• Previous experience in a customer facing role.

Scale AI accelerates the development of AI systems by providing the data, infrastructure, and tooling that power the most advanced models in the world. Our teams operate at the intersection of cutting-edge research, large-scale engineering, and real-world deployment, partnering with leading frontier labs, enterprises, and government agencies to push Generative AI into new capabilities and applications.

As AI rapidly evolves from static models to dynamic, agentic systems, Scale is building the foundational research, evaluation methodologies, and agent/RL infrastructure that will define this next era. You’ll join a high-impact research organization driving advances in large-language models, post-training, evaluation, and agentic/RL environments, helping shape how next-generation AI is built, measured, and deployed.

As a Research Scientist Manager, you will lead a world-class team of research scientists and engineers, define the research roadmap, and drive execution from early prototyping to deployment. You’ll thrive in a fast-moving environment, balancing deep technical leadership with people management, vision setting, and delivery.

You will:

• Lead, mentor and grow a team of research scientists and engineers working on GenAI research initiatives (e.g., evaluation, post-training, agents, RL environments).  
• Define and drive a multi-year research roadmap: identify key scientific questions, set milestones, allocate resources, and ensure rigorous execution.
• Collaborate cross-functionally with engineering, product, client-facing teams and external academic or industry partners to translate research into components, insights, and actionable outcomes.
• Communicate compellingly: publish research, present at conferences, engage in open-source contributions, and represent the team externally.
• Drive an inclusive, high-performing culture: help your team through technical challenges, provide growth opportunities, and attract top talent.
• Stay deeply connected to the research community, understanding major trends, and helping set them.
• Thrive in a high-energy, fast-paced startup environment and are ready to dedicate the time and effort needed to drive impactful results.

Ideally you'd have:

• 5+ years of hands-on research experience (PhD or equivalent preferred) in machine learning, deep learning, generative models, agent/rl systems or related domains.
• A strong track record of research excellence, including publications in top-tier ML/AI venues (NeurIPS, ICML, ICLR, ACL, etc.).
• Experience and track of recording in landing major research impacts in a fast-paced environment
• Experience leading or managing research teams. You’re excited to mentor, coach and develop talent.
• Excellent written and verbal communication skills. You are able to articulate research ideas and outcomes to both technical and non-technical stakeholders.

About Scale

At Scale AI, our mission is to accelerate the development of AI applications. For 8 years, Scale has been the leading AI data foundry, fueling the most exciting advancements in AI, including generative AI, defense applications, and autonomous vehicles. With our recent Series F round, we’re amplifying access to high-quality data to drive progress toward Artificial General Intelligence (AGI). Building on our history of model evaluation with enterprise and government customers, we are expanding our capabilities to set new standards for both public and private evaluations.

About This Role

This role operates at the forefront of AI research and real-world implementation, with a strong focus on reasoning within large language models (LLMs). The ideal candidate will study the data types critical for advancing LLM-based agents, including browser and software engineering (SWE) agents. You will play a key role in shaping Scale’s data strategy by identifying the most effective data sources and methodologies for improving LLM reasoning. Success in this role requires a deep understanding of LLMs, planning algorithms, and novel approaches to agentic reasoning, as well as creativity in tackling challenges related to data generation, model interaction, and evaluation. You will contribute to impactful research on language model reasoning, collaborate with external researchers, and work closely with engineering teams to bring state-of-the-art advancements into scalable, real-world solutions.

Ideally, you’d have:

• Practical experience working with LLMs, with proficiency in frameworks like PyTorch, JAX, or TensorFlow. You should also be skilled at rapidly interpreting research literature and turning new ideas into working prototypes.

• A track record of published research in top ML and NLP venues (e.g., ACL, EMNLP, NAACL, NeurIPS, ICML, ICLR, CoLLM, etc.).

• At least three years of experience solving complex ML challenges, either in a research setting or product development, particularly in areas related to LLM capabilities and reasoning.

• Strong written and verbal communication skills, along with the ability to work effectively across teams.

Nice to have:

• Hands-on experience fine-tuning open-source LLMs or leading bespoke LLM fine-tuning projects using PyTorch/JAX.

• Research and practical experience in building applications and evaluations related to LLM-based agents, including tool-use, text-to-SQL, browser agents, coding agents, and GUI agents.

• Experience with agent frameworks such as OpenHands, Swarm, LangGraph, or similar.

• Familiarity with advanced agentic reasoning techniques such as STaR and PLANSEARCH.

• Proficiency in cloud-based ML development, with experience in AWS or GCP environments.

Our research interviews are designed to assess candidates' ability to prototype and debug ML models, their depth of understanding in research concepts, and their alignment with our organizational culture. We do not conduct LeetCode-style problem-solving assessments.

Scale’s ML platform (RLXF) team builds our internal distributed framework for large language model training and inference. The platform has been powering MLEs, researchers, data scientists and operators for fast and automatic training and evaluation of LLM's, as well as evaluation of data quality.

Scale is uniquely positioned at the heart of the field of AI as an indispensable provider of training and evaluation data and end-to-end solutions for the ML lifecycle. You will work closely across Scale’s ML teams and researchers to build the foundation platform that supports all our ML research and development. You will be building and optimizing the platform to enable our next generation of LLM training, inference and data curation.

If you are excited about shaping the future AI via fundamental innovations, we would love to hear from you!

You will:

• Build, profile and optimize our training and inference framework
• Collaborate with ML teams to accelerate their research and development and enable them to develop the next generation of models and data curation
• Research and integrate state-of-the-art technologies to optimize our ML system

Ideally you’d have:

• Strong excitement about system optimization
• Experience with multi-node LLM training and inference
• Experience with developing large-scale distributed ML systems
• Strong software engineering skills, proficient in frameworks and tools such as CUDA, Pytorch, transformers, flash attention, etc. 
• Strong written and verbal communication skills and the ability to operate in a cross functional team environment

Nice to haves:

• Demonstrated expertise in post-training methods &/or next generation use cases for large language models including instruction tuning, RLHF, tool use, reasoning, agents, and multimodal, etc.

As a Software Engineer on the Machine Learning Infrastructure team, you will build the "Operating System" for our large-scale GPU clusters. You will architect a high-performance training platform that handles the immense complexity of multi-thousand GPU workloads, ensuring every cycle is used efficiently. Your work directly determines the velocity at which our researchers can train and iterate on the world’s most advanced models.

The ideal candidate is a systems expert who thrives on solving the orchestration, networking, and reliability challenges that emerge at massive scale. You will partner closely with researchers to build a seamless, resilient environment that transforms raw compute into breakthrough AI.

You will:

• Architect and scale a multi-tenant orchestration layer that abstracts away the complexity of GPU clusters, ensuring high utilization and seamless job recovery.
• Design and implement scheduling primitives to optimize the lifecycle of training jobs.
• Develop deep observability and automated health-checking into the training stack to proactively identify and isolate hardware failures
• Evaluate and integrate emerging technologies in the CNCF and AI ecosystem (e.g. Ray, Kueue), making data-driven build vs. buy decisions that balance velocity with long-term maintainability.
• Work closely with Finance and Procurement teams to drive our capacity planning process.
• Participate in our team’s on call process to ensure the availability of our services.
• Own projects end-to-end, from requirements, scoping, design, to implementation, in a highly collaborative and cross-functional environment.

Ideally you'd have:

• 5+ years of experience in backend or infrastructure engineering, with at least 2 years focused on orchestrating ML workloads at scale (100+ GPU nodes).
• Strong programming skills in one or more languages (e.g. Python, Go, Rust, C++)
• Experience with complex compute management systems that cover queueing, quotas, preemption, and gang scheduling.
• Experience with distributed training infrastructure, such as EFA, Infiniband, and topology-aware scheduling.
• Experience with distributed storage systems (e.g. Lustre, S3) as they relate to training throughput
• Expert-level knowledge of Kubernetes internals (Custom Resources, Operators, Admission Controllers) and how they interact with device plugins for specialized hardware.
• Familiarity with cloud infrastructure (AWS, GCP) and infrastructure as code (e.g., Terraform).
• Proven ability to solve complex problems and work independently in fast-moving environments.

Nice to haves:

• Experience with distributed training techniques such as DeepSpeed, FSDP, etc.
• Experience with the NVIDIA software and hardware stack (CUDA, NCCL)
• Experience with PyTorch
• Familiarity with post-training algorithms such as GRPO, and with Reinforcement Learning

Scale Labs, Research Scientist — Frontier Risk Evaluations

As the leading data and evaluation partner for frontier AI companies, Scale plays an integral role in understanding the capabilities and safeguarding AI models and systems. Building on this expertise, Scale Labs has launched a new team focused on policy research, to bridge the gap between AI research and global policymakers to make informed, scientific decisions about AI risks and capabilities.

Our research tackles the hardest problems in agent robustness, AI control protocols, and AI risk evaluations to help governments, industry, and the public understand and mitigate AI risk while maximizing AI adoption. This team collaborates broadly across industry, the public sector, and academia and regularly publishes our findings. We are actively seeking talented researchers to join us in shaping this vision.

As a Research Scientist focused on Frontier Risk Evaluations, you will design and create evaluation measures, harnesses and datasets for measuring the risks posed by frontier AI systems. For example, you might do any or all of the following: 

• Design and build harnesses to test AI models and systems (including agents) for dangerous capabilities such as security vulnerability exploitation, CBRN uplift, and other high-risk activities;
• Work with government agencies or other labs to collectively scope and design evaluations to measure and mitigate risks posed by advanced AI systems;
• Publish evaluation methodologies and write technical reports for policymakers.

Ideally you’d have:

• Commitment to our mission of promoting safe, secure, and trustworthy AI deployments in the industry as frontier AI capabilities continue to advance.
• Practical experience conducting technical research collaboratively. You should be comfortable building and instrumenting ML pipelines, writing evaluation harnesses, and quickly turning new ideas from the research literature into working prototypes.
• A track record of published research in machine learning, particularly in generative AI.
• At least three years of experience addressing sophisticated ML problems, whether in a research setting or in product development.
• Strong written and verbal communication skills to operate in a cross-functional team.

Nice to have:

• Experience in crafting evaluations and benchmarks, or a background in data science roles related to LLM technologies.
• Experience with red-teaming or adversarial testing of AI systems.
• Familiarity with AI safety policy frameworks (e.g., NIST AI RMF, EU AI Act, Korea AI Basic Act).

Our research interviews are crafted to assess candidates' skills in practical ML prototyping and debugging, their grasp of research concepts, and their alignment with our organizational culture. We will not ask any LeetCode-style questions. If you’re excited about advancing AI safety and contributing to our mission, we encourage you to apply, even if your experience doesn’t perfectly align with every requirement.

Scale Labs, Research Scientist — AI Controls and Monitoring

As the leading data and evaluation partner for frontier AI companies, Scale plays an integral role in understanding the capabilities and safeguarding AI models and systems. Building on this expertise, Scale Labs has launched a new team focused on policy research, to bridge the gap between AI research and global policymakers to make informed, scientific decisions about AI risks and capabilities.

Our research tackles the hardest problems in agent robustness, AI control protocols, and AI risk evaluations to help governments, industry, and the public understand and mitigate AI risk while maximizing AI adoption. This team collaborates broadly across industry, the public sector, and academia and regularly publishes our findings. We are actively seeking talented researchers to join us in shaping this vision.

As a Research Scientist focused on AI Controls and Monitoring, you will design methods, systems, and experiments to ensure that advanced AI models and agents remain aligned with intended goals, even in high-stakes or adversarial environments. For example, you might:

• Develop monitoring techniques and observability methods that track AI behavior in real time to identify and flag deviations, emergent capabilities, or anomalous outputs;
• Research mechanisms for layered control, including fail-safes, oversight protocols, and intervention methods that can halt or redirect AI systems when risks are detected;
• Design red-team simulations to probe weaknesses in oversight and control mechanisms, and build mitigations to close identified gaps;
• Collaborate with policymakers, engineers, and other researchers to establish standards and benchmarks for AI monitoring and escalation.

Ideally you’d have:

• Commitment to our mission of promoting safe, secure, and trustworthy AI deployments in the industry as frontier AI capabilities continue to advance.
• Practical experience conducting technical research collaboratively. You should be comfortable designing control and monitoring experiments for AI systems, building prototype systems, and quickly turning new ideas from the research literature into working prototypes.
• A track record of published research in machine learning, particularly in generative AI.
• At least three years of experience addressing sophisticated ML problems, whether in a research setting or in product development.
• Strong written and verbal communication skills to operate in a cross-functional team.

Nice to have:

• Experience with runtime monitoring, anomaly detection, or observability for ML systems.
• Familiarity with AI control or alignment research (e.g., scalable oversight, interpretability, debate).
• Experience with post-training and RL techniques such as RLHF, DPO, GRPO, and similar approaches.

Our research interviews are crafted to assess candidates' skills in practical ML prototyping and debugging, their grasp of research concepts, and their alignment with our organizational culture. We will not ask any LeetCode-style questions. If you’re excited about advancing AI safety and contributing to our mission, we encourage you to apply, even if your experience doesn’t perfectly align with every requirement.

As the leading data and evaluation partner for frontier AI companies, Scale is dedicated to advancing the evaluation and benchmarking of large language models (LLMs). We are building industry-leading LLM evals, setting new standards for model performance assessment. Our mission is to develop rigorous, scalable, and fair evaluation methodologies to drive the next generation of AI capabilities.

Our Research teams work with the industry’s leading AI labs to provide high quality data and accelerate progress in GenAI research. As a Staff Machine Learning Research Scientist on the LLM Evals team, you will lead the development of novel evaluation methodologies, metrics, and benchmarks to measure the capabilities and limitations of frontier LLMs. You will help define what "good" looks like in generative AI, driving research that informs both our internal roadmap and the broader research community. This role is critical for designing and executing a roadmap that defines best practices in data driven AI development and will accelerate the next generation of generative AI models in partnership with top foundational model labs.

You will:

• Drive research on the effectiveness and limitations of existing LLM evaluation techniques.
• Design and develop novel evaluation benchmarks for large language models, covering areas such as instruction following, factuality, robustness, and fairness. 
• Communicate, collaborate, and build relationships with clients and peer teams to facilitate cross-functional projects.
• Collaborate with internal teams and external partners to refine metrics and create standardized evaluation protocols.
• Implement scalable and reproducible evaluation pipelines using modern ML frameworks.
• Publish research findings in top-tier AI conferences and contribute to open-source benchmarking initiatives.
• Mentor and guide research scientists and engineers, providing technical leadership across cross-functional projects.
• Stay deeply engaged with the ML research community, tracking emerging work and contributing to the advancement of LLM evaluation science.
• Thrive in a high-energy, fast-paced startup environment and are ready to dedicate the time and effort needed to drive impactful results.

Ideally you'd have:

• 5+ years of hands-on experience in large language model, NLP, and Transformer modeling, in the setting of both research and engineering development
• Experience and track of recording in landing major research impacts in a fast-paced environment
• Experience tech leading a team of research scientists and research engineers
• Excellent written and verbal communication skills
• Published research in areas of machine learning at major conferences (NeurIPS, ICML, ICLR, ACL, EMNLP, CVPR, etc.) and/or journals
• Previous experience in a customer facing role.

Scale's LLM post-training platform team builds our internal distributed framework for large language model training. The platform powers MLEs, researchers, data scientists, and operators for fast and automatic training and evaluation of LLMs. It also serves as the underlying training framework for the data quality evaluation pipeline.

Scale is uniquely positioned at the heart of the field of AI as an indispensable provider of training and evaluation data and end-to-end solutions for the ML lifecycle. You will work closely with Scale’s ML teams and researchers to build the foundation platform which supports all our ML research and development works. You will be building and optimizing the platform to enable our next generation LLM training, inference and data curation.

If you are excited about shaping the future AI via fundamental innovations, we would love to hear from you!

You will:

• Build, profile and optimize our training and inference framework.
• Collaborate with ML and research teams to accelerate their research and development, and enable them to develop the next generation of models and data curation.
• Research and integrate state-of-the-art technologies to optimize our ML system.

Ideally you’d have:

• Passionate about system optimization
• Experience with multi-node LLM training and inference
• Experience with developing large-scale distributed ML systems
• Experience with post-training methods like RLHF/RLVR and related algorithms like PPO/GRPO etc.
• Strong software engineering skills, proficient in frameworks and tools such as CUDA, Pytorch, transformers, flash attention, etc. 
• Strong written and verbal communication skills to operate in a cross functional team environment.

Nice to haves:

• Demonstrated expertise in post-training methods and/or next generation use cases for large language models including instruction tuning, RLHF, tool use, reasoning, agents, and multimodal, etc.

As a Software Engineer on the ML Infrastructure team, you will design and build platforms for scalable, reliable, and efficient serving of LLMs. Our platform powers cutting-edge research and production systems, supporting both internal and external use cases across various environments.

The ideal candidate combines strong ML fundamentals with deep expertise in backend system design. You’ll work in a highly collaborative environment, bridging research and engineering to deliver seamless experiences to our customers and accelerate innovation across the company.

You will:

• Build and maintain fault-tolerant, high-performance systems for serving LLMs workloads at scale.
• Build an internal platform to empower LLM capability discovery.
• Collaborate with researchers and engineers to integrate and optimize models for production and research use cases.
• Conduct architecture and design reviews to uphold best practices in system design and scalability.
• Develop monitoring and observability solutions to ensure system health and performance.
• Lead projects end-to-end, from requirements gathering to implementation, in a cross-functional environment. 

Ideally you'd have:

• 5+ years of experience building large-scale, high-performance backend systems.
• Strong programming skills in one or more languages (e.g., Python, Go, Rust, C++).
• Experience with LLM serving and routing fundamentals (e.g. rate limiting, token streaming, load balancing, budgets, etc.)
• Experience with LLM capabilities and concepts such as reasoning, tool calling, prompt templates, etc.
• Experience with containers and orchestration tools (e.g., Docker, Kubernetes).
• Familiarity with cloud infrastructure (AWS, GCP) and infrastructure as code (e.g., Terraform).
• Proven ability to solve complex problems and work independently in fast-moving environments.

Nice to haves:

• Experience with modern LLM serving frameworks such as vLLM, SGLang, TensorRT-LLM, or text-generation-inference.

About Scale

At Scale AI, our mission is to accelerate the development of AI applications. For 8 years, Scale has been the leading AI data foundry, helping fuel the most exciting advancements in AI, including: generative AI, defense applications, and autonomous vehicles. With our recent Series F round, we’re accelerating the abundance of frontier data to pave the road to Artificial General Intelligence (AGI), and building upon our prior model evaluation work with enterprise customers and governments, to deepen our capabilities and offerings for both public and private evaluations.

About the ACE team 

The Agent Capabilities & Environments (ACE) team, part of Scale’s Research organization, brings together customer-facing Researchers and Applied AI Engineers. Our core mission includes research on agent environments and RL reward signals, benchmarking autonomous agent performance across real-world scenarios and environments, creating robust data programs to improve Large Language Models (LLMs) agentic capabilities and building foundational tools and frameworks for evaluating models as agents. ACE focuses on autonomous agents that dynamically interact with diverse external environments, including code repositories, GUI interfaces, browsers, and more. 

About This Role

This role is at the intersection of cutting-edge AI research and practical application, with a focus on studying the data types essential for building state-of-the-art agents, such as browser and SWE agents. The ideal candidate will explore the data landscape needed to advance intelligent, adaptable AI agents, guiding the data strategy at Scale to drive innovation. This position requires not only expertise in LLM agents and planning algorithms but also creativity in addressing novel challenges related to data, interaction, and evaluation. You will contribute to impactful research publications on agents, collaborate with customer researchers, and work alongside the engineering team to translate these advancements into real-world, scalable solutions.

Ideally you’d have: 

• Practical experience working with LLMs, with proficiency in frameworks like Pytorch, Jax, or Tensorflow. You should also be adept at interpreting research literature and quickly turning new ideas into prototypes.
• A track record of published research in top ML venues (e.g., ACL, EMNLP, NAACL, NeurIPS, ICML, ICLR, COLM, etc.)
• At least three years of experience addressing sophisticated ML problems, either in a research setting or product development.
• Strong written and verbal communication skills and the ability to operate cross-functionally. 

Nice to have:

• Hands-on experience with open source LLM fine-tuning or involvement in bespoke LLM fine-tuning projects using Pytorch/Jax. 
• Hands-on experience and publications in building applications and evaluations related to AI agents such as tool-use, text2SQL, browser agents, coding agents and GUI agents.
• Hands-on experience with agent frameworks such as OpenHands, Swarm, LangGraph, etc.
• Familiarity with agentic reasoning methods such as STaR and PLANSEARCH 
• Experience working with cloud technology stack (eg. AWS or GCP) and developing machine learning models in a cloud environment.

Our research interviews are crafted to assess candidates' skills in practical ML prototyping and debugging, their grasp of research concepts, and their alignment with our organizational culture. We will not ask any LeetCode-style questions.

The Enterprise ML team works on the front lines of the AI revolution, partnering deeply with customers to identify high-impact business problems and build cutting-edge AI systems using Scale’s proprietary research, data, and infrastructure—unlocking domain expertise through high-quality data and expert feedback.

As Director of Enterprise ML, you will lead a world-class team of research scientists and engineers, define the research roadmap, and drive execution from early prototyping to deployment. You’ll thrive in a fast-moving environment, balancing deep technical leadership with people management, vision setting, and delivery.

This role is ideal for a leader who thrives in ambiguity, understands both frontier GenAI capabilities and their limitations, and is motivated by turning research into durable, production-ready systems.

What You’ll Do

• Lead, mentor and grow a team of research scientists and engineers working on GenAI research initiatives (e.g., evaluation, post-training, agents, RL environments).  
• Define and drive a multi-year research roadmap: identify key scientific questions, set milestones, allocate resources, and ensure rigorous execution.
• Collaborate cross-functionally with engineering, product, client-facing teams and external academic or industry partners to translate research into components, insights, and actionable outcomes.
• Communicate compellingly: publish research, present at conferences, engage in open-source contributions, and represent the team externally.
• Drive an inclusive, high-performing culture: help your team through technical challenges, provide growth opportunities, and attract top talent.
• Stay deeply connected to the research community, understanding major trends, and helping set them.
• Thrive in a high-energy, fast-paced startup environment and are ready to dedicate the time and effort needed to drive impactful results.

What We’re Looking For

Core Qualifications

• 8+ years of hands-on research experience (PhD or equivalent preferred) in machine learning, deep learning, generative models, agent/rl systems or related domains.
• A strong track record of research excellence, including publications in top-tier ML/AI venues (NeurIPS, ICML, ICLR, ACL, etc.).
• Experience and track of recording in landing major research impacts in a fast-paced environment
• Experience leading or managing research teams. You’re excited to mentor, coach and develop talent.
• Excellent written and verbal communication skills. You are able to articulate research ideas and outcomes to both technical and non-technical stakeholders.
• Exceptional communication and stakeholder management skills, with the ability to influence executives, customers, and cross-functional partners

Nice to Have

• Hands-on experience building and deploying agent-based, tool-augmented, or workflow-driven LLM systems in enterprise environments
• Prior ownership of enterprise AI platforms, internal ML products, or customer-facing AI services at scale
• Proven track record of partnering directly with enterprises to identify high-impact use cases and deliver measurable business outcomes

As the leading data and evaluation partner for frontier AI companies, Scale is dedicated to advancing the evaluation and benchmarking of large language models (LLMs). We are building industry-leading LLM evals, setting new standards for model performance assessment. Our mission is to develop rigorous, scalable, and fair evaluation methodologies to drive the next generation of AI capabilities.

Our Research teams work with the industry’s leading AI labs to provide high quality data and accelerate progress in GenAI research. As the Tech Lead Manager of the LLM Evals Research team, you will lead a talented team of research scientists and research engineers focused on developing and implementing novel evaluation methodologies, metrics, and benchmarks to assess the capabilities and limitations of our cutting-edge LLMs. This role is critical for designing and executing a roadmap that defines best practices in data driven AI development and will accelerate the next generation of generative AI models in partnership with top foundational model labs. 

You will:

• Lead a team of highly effective research scientists and research engineers on LLM evals.
• Conduct research on the effectiveness and limitations of existing LLM evaluation techniques.
• Design and develop novel evaluation benchmarks for large language models, covering areas such as instruction following, factuality, robustness, and fairness. 
• Communicate, collaborate, and build relationships with clients and peer teams to facilitate cross-functional projects.
• Collaborate with internal teams and external partners to refine metrics and create standardized evaluation protocols.
• Implement scalable and reproducible evaluation pipelines using modern ML frameworks.
• Publish research findings in top-tier AI conferences and contribute to open-source benchmarking initiatives.
• Remain up-to-date on ongoing research in the team, help work through technical challenges, and be involved in design decisions
• Remain deeply involved in the research community, both understanding trends, and setting them
• Thrive in a high-energy, fast-paced startup environment and are ready to dedicate the time and effort needed to drive impactful results.

Ideally you'd have:

• 5+ years of hands-on experience in large language model, NLP, and Transformer modeling, in the setting of both research and engineering development
• Experience and track of recording in landing major research impacts in a fast-paced environment
• Experience supporting and leading a team of research scientists and research engineers
• Excellent written and verbal communication skills
• Published research in areas of machine learning at major conferences (NeurIPS, ICML, ICLR, ACL, EMNLP, CVPR, etc.) and/or journals
• Previous experience in a customer facing role.

About This Role

Lead applied ML engineering on Scale's Applied ML team, powering data infrastructure for leading agentic LLMs (ChatGPT, Gemini, Llama). You will build scalable multi-agent systems to validate agentic reasoning and behaviors, scale human expertise, and drive research into real-world agent reliability failures despite strong benchmarks, shipping production fixes.

Ideal for exceptional engineers with deep research rigor and a relentless focus on practical, high-impact systems. You will iterate rapidly with data, leverage AI tools to accelerate development, and collaborate tightly across engineering, product, and research.

If you excel at turning frontier agent research into reliable deployed systems, we want to hear from you.

You will:

• Build and deploy multi-agent systems for agentic reasoning validation
• Develop pipelines to detect errors and scale human judgment
• Combine classical ML, LLMs, and multi-agent techniques for reliability
• Lead research into agent failure modes and ship fixes
• Use AI tools to speed prototyping and iteration
• Build data-driven evaluations and deploy rapid improvements
• Integrate systems into Scale's platform

Ideally You’ll Have: 

• PhD or MSc in Computer Science, Mathematics, Statistics, or related field
• 3+ years shipping scaled production ML systems
• Demonstrated real-world impact
• Mastery of PyTorch, TensorFlow, JAX, or scikit-learn
• Deep expertise in agentic LLMs and multi-agent systems
• Strong software engineering and microservices (AWS/GCP)
• Rapid, data-driven iteration
• Proficiency using AI tools to accelerate work
• Strong research depth with practical bias
• Excellent cross-functional communication

Nice to Have: 

• Experience prototyping agent evaluation/reliability systems
• Human-in-the-loop or annotation pipeline work
• Open-source contributions in agents, evaluation, or alignment
• Publications on agent reliability (NeurIPS, ICML, ICLR)