Location: This is a 100% remote role open to those who reside in the United States.

About the job 

Testlio's fully managed crowdsourced testing platform, powered by our proprietary intelligence technology - LeoCore™, integrates expert, on-demand testers directly into your release process. Ship faster and more confidently with global coverage across 600,000+ devices, 800+ payment methods, 150+ countries, and 100+ languages. To learn more, visit testlio.com.

We are hiring a visionary and hands-on AI Solutions Architect to own the definition, packaging, technical design, and delivery methodology for our AI testing offerings. This role brings together deep practitioner expertise with a strong business perspective. You will design the testing approach for complex AI-powered applications (including agentic systems), and you will advance Testlio's proprietary framework combining automated and human-in-the-loop evaluation while defining what Testlio sells as a service—setting the catalog, staffing models, and pricing structure. As the technical authority, you will decompose architectures into structured testing approaches and collaborate with sales and delivery to drive our AI business forward.

Why will you love this job?

• Evaluate at a Scale That Exists Nowhere Else: Every vendor has LLM-as-a-judge. Only Testlio can put thousands of vetted experts across 150 countries and 100+ languages behind a grading rubric. You will design evaluation systems that combine model graders and human experts.
• Innovate at the Frontier of QA: Shape the industry standards and operational playbooks for validating agentic AI, large language models, and the data pipelines behind them.
• High-Impact Consultative Visibility: Collaborate with global market leaders across diverse verticals. Interface directly with data scientists, machine learning engineers, and executive sponsors at market-leading global brands to architect their AI quality blueprints.
• Decompose Complex Ecosystems: Turn "unknown unknowns" into structured testing criteria: what the system must do, how it can fail, and what a failure costs.
• Drive company influence: Help build the pipeline, join sales calls and guide delivery on AI Testing opportunities, bringing your expertise to high-impact business decisions.

Why will you love being a part of Testlio?

• Great Culture: Testlio is a female-founded company, and half of our team identifies as women. We’re proud of our inclusive, purpose-driven culture where people genuinely enjoy collaborating. As part of our team (we call ourselves TestLions), you’ll help create exceptional digital experiences for our customers, while also contributing to our freelance network.
• Remote Work: Our culture is built around remote work. We’ve created systems to allow us to successfully work together asynchronously as a fully remote and globally distributed team. Testlio provides the tools and guidance for everyone to succeed in their careers in a fully remote setting. Our working style encourages everyone to make decisions, communicate effectively, and work at a sustainable pace. 
• Investment in You: Your growth and well-being matter to us. You’ll have flexible paid time off—including national holidays, personal days, and sick days—plus stock options so you can grow with Testlio. We also provide a $300 annual learning stipend to support your personal and professional development.
• Winning Business: Testlio is growing, profitable, and cash-strong. We are leading our industry with exceptional clients who provide us with a high NPS score and a 4.7 rating on G2. Our business model is global, enterprise, and subscription-based. Several of our largest clients have been with us for 7+ years, and many spend $500K+/year with Testlio.  

What would your day look like?

Strategic & Commercial Responsibilities

• Define and maintain the AI service catalog: offerings, tiers, packaging, and delivery methodology. Track, analyze, and present key metrics to stakeholder leadership as proof of quality, efficiency, and ROI.
• Vision Activation & Marketing: Imagine, drive, evangelize, and activate a viewpoint on the future of AI Testing. Produce white papers, articles, posts, speeches, and webinars.
• Sales Collaboration: Collaborate with sales and engagement teams to support pre-deal discovery, scoping, and articulating Testlio's AI testing value proposition.
• Offering Lifecycle Management: Use market and competitive intelligence to refine offerings and supply Product Marketing with details for accurate positioning.

Collaboration on Hands-on Delivery

• System & Architectural Deconstruction: Execute comprehensive system-level breakdowns of client AI stack components—including autonomous agents, system prompts, foundation models, and RAG architectures—to formulate a tailored AI testing approach, inform test strategy, design evaluations, and provide technical advisory.
• Delivery Expertise: Act as an escalation point and subject-matter expert for Delivery, stepping in hands-on from scoping to review during complex or new client engagements. 
• Technical Problem Solving: Solve technical roadblocks, stand up environments, and work on solutions that make AI testing and evaluation scalable and repeatable in strategic engagements.
• Framework Advancement: Advance Testlio's proprietary AI testing framework by combining deterministic code graders, LLM-as-a-judge, and human-in-the-loop validation.
• Empower Internal Teams: Develop templates, curated training datasets, and comprehensive documentation to drive seamless, high-quality test execution.

What do you need to succeed?

Technical Skills

• Hands-on Data Science Background: Extensive practical experience developing, training, or fine-tuning machine learning models, NLP structures, or data pipelines.
• AI Evaluation Expertise: Deep familiarity and hands-on experience with modern LLM evaluation frameworks (e.g., DeepEval, RAGAs, etc) and metric-driven validation ecosystems.
• Agentic Framework Proficiency: Direct experience working with, deploying, or testing autonomous agent architectures, planning patterns, or multi-agent swarms (e.g., LangGraph, AutoGen, CrewAI).
• Strong Software Engineering Foundations: Strong software engineering foundations, including proficiency in dealing with complex non-deterministic and deterministic systems.

Human Skills

• Proven Client-Facing Experience: Solid track record successfully leading discovery calls, managing stakeholder workshops, and consulting on real-world enterprise architectures.
• Commercial & Strategic Acumen: Experience designing service offerings, understanding pricing dynamics, and formulating go-to-market strategies within professional services or SaaS.
• Consultative Communication: Exceptional ability to bridge technical data science outputs with clear, strategic customer business value, speaking confidently with both engineers and executives.
• Comfort with Ambiguity: High adaptability to navigate fast-evolving client layouts, unvetted sources, and the non-deterministic characteristics of AI application lifecycles.
• Mentorship Mindset: A passion for continuous learning, documentation, and sharing architectural best practices to uplift internal team members.

What is the application process?

At Testlio, we aim to hire individuals who are excited about their role, thrive in a fully remote environment, and have strong long-term potential with our team. Because we are a fully distributed company, our interview process includes conversations with several team members so you can get a well-rounded understanding of the role, the people you’ll work with, and how we collaborate. As a result, our interview process typically takes 3–4 weeks to complete. 

Interview Process:

• Application
• Recruiter interview
• TestGorilla assessment
• ~ 4 Team and Stakeholder interviews
• Reference checks 
• Offer & background check

Diversity and Inclusion

Testlio is an equal-opportunity employer deeply committed to creating an inclusive environment for people of all backgrounds and identities. We are female-founded, and half of our team members identify as women. For more information, see the DEI section of our website.

#LI-Remote