About this Data Quality and Labeling Specialist role at Collate
We're looking for a Data Quality and Labeling Specialist to bring life-sciences expertise to the data behind our AI. Our models draft, extract from, and review regulated documents across drug, diagnostic, and medical device development, and this role makes sure the data they learn from and are measured against is accurate, consistent, and grounded in how regulated work is actually done.
This is a hands-on role at the intersection of domain expertise and data quality. You'll work side by side with our AI researchers and engineers, and your judgment about what a correct document looks like will shape the product directly.
What You’ll Do
Review and label regulatory, quality, and clinical documents and model-generated outputs for accuracy, completeness, and compliance.
Build and maintain gold-standard datasets used to train and evaluate our models.
Write and maintain labeling guidelines and evaluation rubrics as new document types and use cases are added.
Run calibration with other reviewers to keep labels consistent and resolve disagreements.
Audit labeled data before it reaches training and evaluation, and track label quality over time.
Turn recurring model errors into clear, actionable feedback for the research and engineering teams.
What We’re Looking For
Experience in a regulated life-sciences function such as regulatory affairs, quality, clinical, or CMC, at a pharma, biotech, diagnostics, or medical device company.
Familiarity with the documents and standards that govern that work (e.g., FDA, EMA, ICH, or ISO requirements).
Strong attention to detail: you notice an unsupported claim or inconsistent value and can explain why it matters.
Ability to apply structured rubrics consistently and to flag when a rubric itself needs changing.
Clear written communication.
Prior experience with data annotation, model evaluation, or writing guidelines for other reviewers is a plus.
Excitement to work in an early-stage environment on-site in San Francisco.
Why Join Collate?
Impact: The documents you help get right are the ones that put drugs, diagnostics, and devices in front of patients sooner.
Ownership: You define what "correct" means for a new category of AI product and own the data that enforces it, from day one.
Learning: Work daily with AI researchers, engineers, and life-sciences leaders; see how your labels change model behavior within the week.
Upside: Join early enough for meaningful equity and a role that grows with the data function—into leading a team of reviewers and contract SMEs as we scale.
