Run short, instrumented spikes and benchmarking on new models, tools and frameworks: LLMs, agentic systems, vector databases, orchestration frameworks, copilots, assistants and more.

Compare vendor and open-source options, documenting trade-offs across quality, cost, latency, security and integration complexity.

Deliver concise decision memos with clear recommendations: adopt, watch, or avoid.

Design and maintain evaluation environments (e.g. datasets, prompts, scenarios, telemetry) to test models under realistic constraints.

Build automation and tooling to measure quality, robustness, latency and cost, including regression tracking over time.

Ensure every evaluated technology has benchmark coverage and a documented risk and limitations view.

Build enough of a system to understand how a technology behaves under realistic conditions, not just in vendor demos or isolated examples.

Explore architecture, integration patterns, operational constraints, security boundaries, and failure modes through working prototypes.

Determine what must be true for a proof of concept to become a viable production capability.

Prefer focused prototypes that answer specific technical questions over prematurely building production systems.

Translate technical findings into clear decision memos for both technical and non-technical stakeholders.

For validated technologies, produce readiness guidance covering recommended patterns, guardrails, known limitations, operational considerations, and integration requirements.

Coordinate hand-offs to the engineering teams responsible for productionization.

Support those teams during the transition when deep context from the evaluation is required, without becoming the permanent owner of the resulting system.

Track what happens after Lab recommendations and use those outcomes to improve future evaluation methods.

Work with Security, Legal, Compliance and other AI teams to document risk assessments, mitigations and governance recommendations for each evaluated technology.

Maintain checklists, decision templates and lightweight standards reusable across evaluations and by partner teams.

Incorporate learnings from third-party AI tooling already in use, such as external copilots and the AWS AI suite, into adoption guidelines.

Partner with other AI teams and domain teams to ensure clear boundaries and smooth collaboration.

Participate in hiring as a technical evaluator and culture champion.

Mentor engineers in the Lab and adjacent teams on evaluation methods, benchmarking and experimental design.

Share knowledge through internal write-ups, tech talks and occasional external meetups and conferences.

8+ years of software engineering experience, including significant experience operating at senior or Staff-level scope.

Deep hands-on experience building and evaluating systems based on LLMs and modern AI tooling.

Strong software engineering fundamentals and the ability to rapidly build high-quality experimental systems.

Experience building agentic or multi-step AI systems involving tool use, orchestration, state, retrieval, or external integrations.

Strong knowledge of cloud infrastructure, preferably AWS, and the ability to run experimental workloads securely and cost-consciously.

Experience with observability, telemetry, testing, and benchmarking of complex systems.

Ability to reason about system architecture, reliability, scalability, asynchronous workflows, and distributed components where relevant.

Track record of designing experiments or benchmarks that influenced meaningful technical decisions.

Track record designing and running benchmarks that compare AI models and tools under real constraints.

Experience constructing evaluation datasets: task selection, labelling, holdout discipline, and keeping a set useful as models improve.

Working knowledge of LLM-as-judge methods and their failure modes, alongside human evaluation, inter-annotator agreement, and a view on when each is appropriate.

Able to reason about statistical significance on small samples, and to state confidence honestly rather than over-reading a result.

Familiarity with regression tracking, telemetry and versioning, so that a result stays reproducible months later.

Able to turn ambiguous "we should try this new thing" ideas into well-scoped evaluation plans with clear hypotheses and metrics.

Comfortable making trade-off calls across quality, latency, cost and vendor lock-in, and documenting them clearly.

Experience writing short, opinionated decision memos that help others move fast.

Can explain technical results to non-specialists in concrete, concise terms.

Experience working with platform, product and operations teams to align evaluations with real use cases.

Able to influence without authority, aligning teams around shared standards and guardrails.

Curious and biased toward experimentation, combined with disciplined measurement and risk awareness.

Comfortable in a small, high-leverage team without embedded PMs. You structure your own work and keep stakeholders informed.

Builder attitude: you prefer reusable tools, templates and playbooks over one-off work.