Job Posting Title:

Req ID:

Job Description:

Disney Entertainment and ESPN Product & Technology

Technology is at the heart of Disney’s past, present, and future. Disney Entertainment and ESPN Product & Technology is a global organization of engineers, product developers, designers, technologists, data scientists, and more – all working to build and advance the technological backbone for Disney’s media business globally.

The team marries technology with creativity to build world-class products, enhance storytelling, and drive velocity, innovation, and scalability for our businesses. We are Storytellers and Innovators. Creators and Builders. Entertainers and Engineers. We work with every part of The Walt Disney Company’s media portfolio to advance the technological foundation and consumer media touch points serving millions of people around the world.

Here are a few reasons why we think you’d love working here:

1. Building the future of Disney’s media: Our Technologists are designing and building the products and platforms that will power our media, advertising, and distribution businesses for years to come.

2. Reach, Scale & Impact: More than ever, Disney’s technology and products serve as a signature doorway for fans' connections with the company’s brands and stories. Disney+. Hulu. ESPN. ABC. ABC News…and many more. These products and brands – and the unmatched stories, storytellers, and events they carry – matter to millions of people globally.

3. Innovation: We develop and implement groundbreaking products and techniques that shape industry norms and solve complex and distinctive technical problems.

Ad Platforms is responsible for Disney’s industry-leading ad technology and products – driving advertising performance, innovation, and value in Disney’s sports, news, and entertainment content, across all media platforms.

Job Summary

We are hiring a Sr Product Development Engineer I to support Model Evaluation & Engineering Systems within Foundations Engineering – AI Production Engineering.

In this role, you will design and implement automated benchmarking workflows, regression detection pipelines, and evaluation harnesses for AI-powered services. You will work closely with the Staff-level lead to operationalize model quality standards and integrate evaluation workflows into release processes.

This role is hands-on and execution-focused, contributing directly to measurable AI quality control and model promotion validation.

Responsibilities:

Evaluation Pipeline Development

• Implement automated benchmarking workflows for model versions and providers.

• Build regression detection systems comparing model outputs across updates.

• Develop evaluation harnesses for structured test datasets.

• Integrate evaluation results into CI/CD release workflows.

Model Quality & Testing Systems

• Support A/B testing systems for model comparison.

• Contribute to response consistency and hallucination analysis tooling.

• Assist in implementing prompt and workflow drift detection automation.

• Ensure reproducibility and traceability of evaluation results.

Collaboration & Execution

• Partner with AI Core Engineering to validate new model integrations.

• Work with Runtime & Reliability teams to correlate evaluation results with production performance.

• Participate in model readiness reviews and certification processes.

Basic Qualifications:

• 5+ years of backend or systems engineering experience.

• Strong proficiency in Python.

• Experience building automated testing or evaluation pipelines.

• Experience integrating external APIs or model services.

• Experience working in cloud-based environments.

Preferred Qualifications:

• Exposure to AI-powered systems or LLM integrations.

• Experience with benchmarking or structured dataset validation systems.

• Familiarity with CI/CD-based release gating workflows.

• Experience contributing to experimentation or validation platforms.

Experience with:

• Backend services and distributed systems.

• Automated testing frameworks and regression detection systems.

• Evaluation dataset processing and validation tooling.

• CI/CD pipelines and automated release gating.

• Observability or analytics systems for performance measurement.

• LLM APIs or model-based services.

• Cloud-native deployment environments

Required Education:

• Bachelor’s degree in Computer Science, Engineering, or related technical field (Master’s preferred), or equivalent experience

Job Posting Segment:

Job Posting Primary Business:

Primary Job Posting Category:

Employment Type:

Primary City, State, Region, Postal Code:

Alternate City, State, Region, Postal Code:

Date Posted: