Roles & Responsibilities 

Generative AI & LLM Evaluation 

• Build automated testing suites to detect hallucinations, bias, toxicity, and prompt injection vulnerabilities across LLM-powered products 

• Implement automated evaluations for RAG systems measuring context relevance, groundedness, and answer faithfulness using frameworks like RAGAS or DeepEval 

• Design test beds to validate multi-agent workflows — tool-calling accuracy, multi-step reasoning, memory, and autonomous decision loops 

• Build and run automated conversation simulations — scripted and synthetic user journeys — to stress-test agent behaviour across intents, edge cases, and multi-turn dialog flows 

• Create prompt regression frameworks to assess how changes in system prompts, temperature, and sampling parameters impact output consistency 

Data Quality Assurance 

• Statistically validate AI data outputs — distributions, precision/recall, error pattern analysis — to catch silent data quality failures before production 

• Programmatically audit data ingestion, transformation, and feature store pipelines for schema drift and data corruption 

• Validate vector DB indexing, embedding semantic similarity accuracy, and retrieval latency 

• Verify quality, diversity, and privacy compliance of synthetic datasets used for model training and evaluation 

Classical ML & Deep Learning Validation 

• Maintain automated suites tracking ML metrics — Precision, Recall, F1, ROC-AUC — and deep learning loss curves across model versions 

• Implement continuous monitoring scripts to detect data and concept drift on live inference endpoints 

Automation Engineering & CI/CD 

• Build and maintain scalable test automation frameworks for APIs, backend services, and model endpoints 

• Embed AI evaluation and data QA suites into MLOps and CI/CD pipelines so quality failures block releases automatically 

• Define and track AI quality KPIs and communicate release readiness to engineering and product teams 

Experience of 5+ years SDET role 

Technical Skills & Frameworks 

Core Programming 

• Python — expert level; test automation, eval pipelines, data analysis (Pandas, NumPy, Pytest) 

• SQL — data output validation, ground truth querying, pipeline data quality checks 

GenAI & Evaluation 

• RAGAS / TruLens / DeepEval / Promptflow etc — LLM evaluation frameworks for measuring faithfulness, hallucination rate, and task success 

• LangChain / LangSmith / LlamaIndex — agent workflow testing, prompt tracing, and LLM response debugging 

• OpenAI / Anthropic / Hugging Face APIs — direct LLM endpoint testing and output consistency validation 

• Vector DBs — retrieval quality testing, embedding validation, and latency benchmarking 

• Pandas / NumPy etc. — statistical analysis for output validation and error pattern investigation, data profiling, schema validation, and pipeline integrity checks 

API & Automation 

• Pytest — modular, reusable test framework for AI eval and automation suites 

• Postman / REST Assured / Requests — API contract validation and service-level integration testing 

MLOps & CI/CD 

• MLflow — tracking model versions and eval runs to detect regressions across updates 

• Docker / GitHub Actions / Jenkins — containerised test environments and deployment pipeline automation 

Observability 

• Grafana / Kibana / OpenTelemetry — monitoring AI system health, output drift, and distributed tracing across agent pipelines 

Good to Have 

• Cloud AI Services — AWS Bedrock, Azure OpenAI, or GCP Vertex AI for testing managed model endpoints and cloud-deployed agents 

• MLOps Platforms — MLflow, Kubeflow, Weights & Biases, or Feast feature stores for experiment tracking and model governance 

• ML Frameworks — Scikit-learn, TensorFlow, or PyTorch familiarity for understanding model internals and validating training pipelines 

• Infrastructure as Code — Docker, Kubernetes, Terraform for managing containerised test environments at scale 

• UI Automation — Playwright or Cypress for end-to-end conversational AI application testing 

• Performance Engineering — Locust or JMeter for load testing heavy AI inference endpoints under peak traffic 

• Statistical Hypothesis Testing — t-tests, confidence intervals, significance testing to distinguish real quality signal from noise 

• Synthetic Data Generation — using LLMs to generate diverse test cases and evaluation datasets at scale