About this Senior Software Engineer, Test role at Roku
Teamwork makes the stream work.
Roku is changing how the world watches TV
Roku is the #1 TV streaming platform in the U.S., Canada, and Mexico, and we've set our sights on powering every television in the world. Roku pioneered streaming to the TV. Our mission is to be the TV streaming platform that connects the entire TV ecosystem. We connect consumers to the content they love, enable content publishers to build and monetize large audiences, and provide advertisers unique capabilities to engage consumers.
From your first day at Roku, you'll make a valuable - and valued - contribution. We're a fast-growing public company where no one is a bystander. We offer you the opportunity to delight millions of TV streamers around the world while gaining meaningful experience across a variety of disciplines.
What does the team work on?
Our Vertical Search team's mission is to deliver the best TV search experience: fast, comprehensive, and reliably relevant. For REA Vertical Search, that means helping users discover the right content across typed, voice, and assistant-driven experiences while supporting a growing set of subscription services and Roku Channel discovery surfaces. We partner with search engineers, ML engineers, product managers, and data partners to define quality strategy for relevance-sensitive experiences, build scalable automation and evaluation frameworks, and apply agentic automation to keep pace with expanding catalog breadth and use cases.
What is the role?
We are looking for a Lead, Search Relevance Quality to work closely with the Roku Vertical Search Team to define, measure, and improve Search and Assistant relevance quality. This person will lead quality strategy for relevance-sensitive experiences, build scalable automation and evaluation frameworks, and apply agentic automation to keep pace with expanding catalog breadth and use cases. The role requires balancing automated quality systems with pragmatic human-in-the-loop review where needed until confidence and coverage are high. Quality for Vertical Search will be measured with clear, trusted metrics rather than inferred only from downstream engagement signals. Search and Assistant launches will have reusable relevance-evaluation coverage, stronger release confidence, and faster debugging loops. The team can scale quality across a growing number of subscription services and Roku Channel experiences without linear growth in manual effort.
How will I use AI at Roku?
At Roku, AI agents do most of the keystroke-level coding. Engineers act as the technical leads for those agents, setting context, planning work, verifying outputs, and recovering when the agent goes off course. If you are excited about working this way, we want to talk to you.
At Roku, we're embracing AI as a powerful tool to amplify human creativity, accelerate innovation, and deliver better results for our customers. Across teams, we're exploring how AI can enhance our work and we're only looking for people who are curious, adaptable, and excited to grow alongside these technologies.
What are the responsibilities of the role?
- Define the quality strategy for Vertical Search and Assistant relevance across Roku Premium subscription services, Roku Channel search journeys, and related discovery experiences
- Build scalable frameworks to measure search quality across query understanding, retrieval, ranking, result presentation, and assistant response quality
- Create automated evaluation pipelines for offline, pre-launch, and continuous post-launch quality monitoring using modern test infrastructure and agentic workflows
- Partner with search engineers, ML engineers, product managers, and data partners to define KPI hypotheses, guardrails, and learning plans for quality-related launches and experiments
- Develop quality signals and gold-standard datasets for vertical search, including subscription-service discovery, entity matching, ranking relevance, and assistant answer quality
- Use agentic automation to generate, maintain, and expand test coverage and evaluation assets while ensuring governance, traceability, and reliability of the resulting workflows
- Introduce the right mix of manual review and automation for ambiguous or high-impact experiences until the automated system is proven and trusted
- Drive root-cause analysis for relevance regressions and quality escapes, and convert findings into durable frameworks, dashboards, and release gates
- Lead end-to-end test and quality architecture for relevance-heavy search and assistant systems
- Design evaluation methodologies for search result quality, assistant response quality, and discovery experience consistency
- Build APIs, tools, datasets, and automation harnesses for offline and online quality validation
- Establish CI/CD quality gates for ranking, retrieval, and assistant-related regressions
- Partner with engineering to instrument systems for observability, diagnosis, and actionable quality telemetry
- Translate vague quality concerns into measurable test plans, labeled datasets, and scalable automation workflows
- Mentor engineers on quality engineering best practices and raise the technical bar for automation design, signal quality, and testability
What experience would help someone be successful in this role at Roku?
- 8+ years in relevance quality engineering, or software engineering roles with significant ownership of automation and system quality
- Strong coding skills in Python, Java, or similar languages, with the ability to build production-grade test frameworks and internal tooling
- Deep experience with API, integration, service, and UI automation in complex distributed systems
- Experience defining test strategy, test design, and quality architecture for large-scale products
- Strong analytical ability to define metrics, interpret results, and isolate root causes across data, ranking logic, and system behavior
- Demonstrated ability to work cross-functionally with engineering, product, and data stakeholders
- Experience introducing scalable automation in environments where quality must be measured, not assumed
- Experience with search relevance, ranking systems, information retrieval, recommendations, NLP, ASR/NLU, or LLM/assistant evaluation
- Experience building quality frameworks for user-intent-heavy products where correctness is subjective and must be inferred through labels, rubrics, or review processes
- Familiarity with agentic AI tooling for test generation, evaluation orchestration, or QA workflow automation
- Experience supporting experimentation-driven teams where every meaningful change has KPI hypotheses and guardrails
- Experience with subscription, media, streaming, or catalog discovery products
- You have built fluency across the agentic engineering toolchain — coding harnesses like Claude Code or Cursor, MCP servers, custom skills, or agent frameworks, and you can describe projects where you shipped real work with these tools; you know how to drive an agent, verify its output, and ramp on an unfamiliar codebase with an agent helping you
#LI-DN1
What's Roku's approach to hybrid working?
Roku fosters an inclusive and collaborative environment where teams generally work in the office Monday through Thursday. Fridays are generally flexible for remote work, except for employees whose specific roles or assigned office location require five days' a week attendance.
What are some of the benefits?
Roku is committed to offering a diverse range of benefits as part of our compensation package to support our employees and their families. Our comprehensive benefits include global access to mental health and financial wellness support and resources. Local benefits include statutory and voluntary benefits which may include healthcare (medical, dental, and vision), life, accident, disability, commuter, and retirement options (401(k)/pension). Employees are supported in taking time off, in accordance with local leave policies and other personal needs to support their evolving work and life needs. It's important to note that not every benefit is available in all locations or for every role. For details specific to your location, please consult with your recruiter.
Accommodations
Roku welcomes applicants of all backgrounds and provides reasonable accommodations and adjustments in accordance with applicable law. If you require reasonable accommodation at any point in the hiring process, please direct your inquiries to [email protected].
What should I know about Roku's culture?
Roku is a great place for people who want to work in a fast-paced environment where everyone is focused on the company's success rather than their own. We try to surround ourselves with people who are great at their jobs, who are easy to work with, and who keep their egos in check. We appreciate a sense of humor. We believe a fewer number of very talented folks can do more for less cost than a larger number of less talented teams. We're independent thinkers with big ideas who act boldly, move fast and accomplish extraordinary things through collaboration and trust. In short, at Roku you'll be part of a company that's changing how the world watches TV.
We have a unique culture that we are proud of. We think of ourselves primarily as problem-solvers, which itself is a two-part idea. We come up with the solution, but the solution isn't real until it is built and delivered to the customer. That penchant for action gives us a pragmatic approach to innovation, one that has served us well since 2002.
To learn more about Roku, our global footprint, and how we've grown, visit https://www.weareroku.com/factsheet.
By providing your information, you acknowledge that you want Roku to contact you about job roles, that you have read Roku's Applicant Privacy Notice, and understand that Roku will use your information as described in that notice. If you do not wish to receive any communications from Roku regarding this role or similar roles in the future, you may unsubscribe at any time by emailing [email protected].
