Overview

Black Canyon Consulting (BCC) is seeking highly qualified Medical Scientific Annotation Specialists to support a critical NIH research initiative focused on modeling sepsis progression using multimodal clinical data from MIMIC-IV. 

This role will support the development of a high-quality annotated clinical reference set by reviewing and refining clinical events and their timing within patient records. Annotators will work with clinical notes and, where applicable, structured electronic health record (EHR) data to determine when clinically meaningful events occurred and assess the certainty of those timestamps. 

The project will also evaluate the use of large language models (LLMs) to assist with clinical annotation. Ideal candidates will have a strong clinical background, experience interpreting complex patient records, and familiarity with sepsis and critical care. 

Purpose 

The primary aim of this project is to improve the temporal characterization of sepsis progression by creating a certainty-aware annotated reference set that can be used to evaluate unimodal and multimodal approaches to clinical timeline construction. 

Annotators will help identify and validate clinically meaningful events, determine their timing, distinguish certain from uncertain timestamps, and assess the clinical evidence supporting event timing.

Roles & Responsibilities

• Training & Onboarding 

  Participate in remote project training and onboarding, including: 

  • Annotation guidelines and protocols. 
  • Clinical event and timestamp annotation procedures. 
  • Use of the project's annotation and visualization tools. 
  • Required CITI training and secure environment access. 
  • Calibration exercises prior to production annotation. 

   

• Clinical Annotation 

   

  • Review clinical documentation from patients with sepsis and identify clinically meaningful events related to disease progression and patient care. 

  •  

     

    Review and correct draft events and candidate timestamps generated through the project's annotation workflow. 

     

  • Apply the project's defined event schema consistently across assigned records. 

     

• Temporal Event Validation 

   

  • Determine when clinical events occurred based on available documentation. 

     

  • Classify timestamps as certain when supported by an explicit time or unambiguous temporal anchor. 

     

  • Identify uncertain timestamps when timing must be inferred from clinical context, judgment, or supporting structured data. 

     

  • Resolve temporal expressions such as "overnight," "earlier today," or "worsening over several hours" using available evidence. 

     

• Multimodal Clinical Review 

   

  • Review structured EHR information, when provided, to support or refine the timing of events described in clinical notes. 

     

  • Evaluate relevant clinical signals such as vital signs, laboratory trends, interventions, vasopressor initiation or escalation, urine output, ventilator settings, and other indicators of patient deterioration. 

     

  • Document whether timing evidence is supported by notes only, structured data only, or a combination of notes and structured data. 

     

• Quality Control & Adjudication 

   

  • Participate in pilot calibration and double-annotation activities. 

     

  • Review annotation disagreements and participate in adjudication as needed. 

     

  • Follow versioned annotation guidelines and incorporate updates resulting from calibration and adjudication. 

     

  • Maintain consistent annotation quality across production batches. 

     

  • Support targeted quality assurance for uncertain timestamps, discrepancies between narrative and structured data, and high-risk clinical transitions. 

     

• Communication 

   

  • Participate in regular project meetings and annotation discussions. 

     

  • Raise ambiguous cases or guideline questions for team review. 

     

  • Incorporate feedback and guideline updates throughout the annotation effort. 

     

  • Collaborate with other annotators and project team members to maintain consistency across the reference dataset. 

Required Skills 

• Clinical training and demonstrated ability to interpret complex patient records. 
• Familiarity with sepsis, acute organ dysfunction, and clinical deterioration. 
• Ability to interpret clinical notes, vital signs, laboratory results, interventions, and other EHR information. 
• Strong clinical reasoning and attention to detail. 
• Ability to interpret temporal information within clinical documentation. 
• Ability to consistently apply detailed annotation guidelines. 
• Comfort working with digital annotation tools and structured clinical data. 
• Ability to work independently and collaboratively in a distributed team environment. 

Education & Experience 

Required 

• Advanced degree or professional certification in a clinical field, such as MD, DO, PA, NP, or RN with relevant clinical experience. 
• Experience reviewing and interpreting clinical documentation and patient records. 

Benefits and Salary

We attract the best people in the business with our competitive benefits package that includes medical, dental and vision coverage, 401k plan with employer contribution, paid holidays, vacation, and tuition reimbursement.

We offer a competitive salary commensurate with experience and location.

If you enjoy being a part of a high performing, professional service and technology focused organization, please apply today!