Senior Machine Learning Engineer - Model Evaluations, Public Sector

The Public Sector ML team at Scale deploys advanced AI systems—including LLMs, agentic models, and multimodal pipelines—into mission-critical government environments. We build evaluation frameworks that ensure these models operate reliably, safely, and effectively under real-world constraints. As an ML Engineer, you will design, implement, and scale automated evaluation pipelines that help customers trust and operationalize advanced AI systems across defense, intelligence, and federal missions.

You will:

• Develop and maintain automated evaluation pipelines for ML models across functional, performance, robustness, and safety metrics, including LLM-judge–based evaluations.
• Design test datasets and benchmarks to measure generalization, bias, explainability, and failure modes.
• Build evaluation frameworks for LLM agents, including infrastructure for scenario-based and environment-based testing.
• Conduct comparative analyses of model architectures, training procedures, and evaluation outcomes.
• Implement tools for continuous monitoring, regression testing, and quality assurance for ML systems.
• Design and execute stress tests and red-teaming workflows to uncover vulnerabilities and edge cases.
• Collaborate with operations teams and subject matter experts to produce high-quality evaluation datasets.
• Comfortable with light travel (approximately 10%) for customer interaction and team needs.

This role will require an active security clearance or the ability to obtain a security clearance.

Ideally you’d have:

• Experience in computer vision, deep learning, reinforcement learning, or NLP in production settings.
• Strong programming skills in Python; experience with TensorFlow or PyTorch.
• Background in algorithms, data structures, and object-oriented programming.
• Experience with LLM pipelines, simulation environments, or automated evaluation systems.
• Ability to convert research insights into measurable evaluation criteria.

Nice to haves:

• Graduate degree in CS, ML, or AI.
• Cloud experience (AWS, GCP) and model deployment experience.
• Experience with LLM evaluation, CV robustness, or RL validation.
• Knowledge of interpretability, adversarial robustness, or AI safety frameworks.
• Familiarity with ML evaluation frameworks and agentic model design.
• Experience in regulated, classified, or mission-critical ML domains.

WHO WE ARE:  

Aviatrix® is pioneering the Cloud Native Security Fabric — the architecture the Containment Era requires. The Cloud Native Security Fabric governs every workload communication path across every cloud, every VPC, every Kubernetes cluster, and every serverless function, from a single policy plane. One rule. Universal propagation. Enforced at the workload, not at a chokepoint. Trusted by more than 500 of the world's leading enterprises. For more information, visit aviatrix.ai. 

ABOUT THE ROLE:

We are looking for a seasoned Senior DevOps Engineer to join our team and take ownership of our CI/CD ecosystem, cloud infrastructure automation, and developer platform. This is a journey-level, fully qualified position where you will independently design and drive solutions across complex, diverse projects — bringing both technical depth and collaborative leadership to everything you do.

You will work at the intersection of infrastructure engineering, AI-driven tooling, and modern cloud operations, partnering closely with development and TechOps teams to accelerate delivery cycles and ensure the reliability, security, and scalability of our systems.

Key Responsibilities:

• Design, implement, and maintain robust, scalable CI/CD pipelines across multiple environments, applying industry best practices to optimize delivery speed and system reliability.
• Automate the provisioning and lifecycle management of cloud resources (VMs, EC2, and more) using Infrastructure as Code — with Terraform as the primary tool — ensuring consistent, repeatable, and auditable deployments.
• Manage and optimize cloud networking configurations, supporting complex developer workflows and infrastructure scalability requirements.
• Integrate AI and large language model (LLM) capabilities into DevOps tooling and developer workflows, including leveraging Model Context Protocol (MCP) for intelligent automation and context-aware pipeline tooling.
• Collaborate with engineering and operations teams to identify bottlenecks, reduce toil, and continuously improve the release process.
• Apply security best practices to deployment pipelines and cloud infrastructure, proactively identifying and remediating vulnerabilities.
• Monitor, observe, and optimize infrastructure performance, taking initiative to resolve complex issues before they impact customers.
• Evaluate emerging tools and technologies — particularly in the AI/LLM space — and provide guidance on how to incorporate them effectively into organization workflows.
• Communicate technical strategies and recommendations clearly to senior internal and external stakeholders, adapting your approach to diverse audiences and often driving alignment on difficult decisions.

Qualifications:

• 5+ years of related DevOps/infrastructure engineering experience (or 3 years with a Master’s degree or equivalent experience).
• Deep, hands-on expertise with Terraform for infrastructure as code — this is a must-have.
• Hands-on, production-level experience building and deploying AI/LLM-integrated systems — this is a must-have. You have worked directly with large language model APIs, designed AI-assisted workflows, and shipped real tooling powered by LLMs in an engineering or DevOps context.
• Hands-on, production-level experience with Model Context Protocol (MCP) — this is a must-have. You have implemented MCP-based integrations to build intelligent, context-aware developer tools, automation pipelines, or agent workflows, not just read about it.
• Strong command of CI/CD platforms (e.g., GitHub Actions, GitLab CI, Jenkins, CircleCI, or similar).
• Experience provisioning and managing cloud infrastructure on AWS, GCP, or Azure (EC2, VMs, VPCs, IAM, etc.).
• Solid understanding of cloud networking concepts (VPCs, subnets, security groups, load balancers, DNS).
• Demonstrated ability to work independently, resolve ambiguous and complex problems, and deliver high-quality solutions with minimal supervision.

Preferred:

• Experience with Kubernetes, Helm, or container orchestration platforms.
• Familiarity with GitOps workflows (e.g., ArgoCD, Flux).
• Scripting proficiency in Python, Bash, or Go.
• Knowledge of observability tooling (Datadog, Prometheus/Grafana, OpenTelemetry).
• Experience building internal developer platforms (IDPs).

US Pay Range:

The US National annual base salary range for this full-time position is $131,500-$154,700 + benefits + 401(k) match + equity. The pay range is determined by the role, work location, job-related skills, level, experience and relevant education. [Certain roles are eligible to earn sales commission, depending on the terms of the applicable plan.] The range displayed is the minimum and maximum target base salary and is applicable only for new hires for the listed position located in the US. Your Talent Advisor can share more details regarding salary ranges, benefits, and equity for your location during the hiring process.

About PLACE

PLACE is shaping the future of real estate. We are revolutionizing the real estate industry with advanced technology and innovative strategies. Our intuitive platform simplifies the real estate process, providing clients with the tools and insights needed to excel.

About Transactly

Transactly is now proudly a PLACE-owned company. At Transactly, we are building the future of real estate services. Through our Connect Platform, we connect homebuyers and renters with the services they need right away, including internet, home security, home warranties, moving services, and more.

What You'll Love About This Role

• Competitive compensation commensurate with experience
• 100% remote flexibility with direct access to and visibility with leadership
• A role with real ownership: you won't just manage work, you'll shape how the company builds and grows
• The chance to spearhead AI adoption and innovation at a fast-moving company
• Great benefits, including:
  • 401(k) with company match
  • Medical, dental, and vision insurance
  • Stock purchase program

What This Role Really Is

This is a driver role. You’ll report directly to the CEO, own the product roadmap end to end and be accountable for shipping efficient solutions. 

You’ll be the internal authority on where and how agentic AI can replace, augment, or accelerate our operational and sales processes. We don’t need someone who’s “AI-curious.” We need someone who understands models, agent architectures, and automation tooling well enough to identify high-leverage opportunities and execute on them.

If you like owning things, driving momentum, and making sure the right work gets done at the right time, this role was built for you.

What You’ll Own

Product Ownership

• Own the product roadmap: define priorities, manage the backlog, run sprints, and hold the development team accountable to delivery timelines.
• Drive our website rebuild from requirements through launch, making sure it ships on time and reflects our product positioning.
• Own API integration projects — define requirements, coordinate with internal and external teams, and ensure delivery.
• Manage our HubSpot implementation and hold the CRM/marketing automation team accountable for outcomes, not just activity.
• Translate business objectives into clear technical requirements. Be the bridge between leadership’s vision and what actually gets built.
• Keep leadership informed with signal, not noise. Surface risks early, remove blockers, and protect the team’s focus.

AI & Automation

• Serve as the company’s internal AI strategist. Continuously evaluate our operational and sales workflows to identify processes ripe for automation with agentic AI.
• Architect and deploy AI agent workflows that reduce manual effort, accelerate cycle times, or eliminate handoff friction across the business.
• Stay current on the AI tooling landscape — models, agent frameworks, integration platforms — and make pragmatic build-vs-buy decisions.
• Run pilots with clear success criteria, measure ROI, and scale what works. Kill what doesn’t.
• Build internal capability: help the broader team understand what’s possible with AI and drive adoption beyond just the tech org.

Who You Are

• A driver. You don’t wait for permission or consensus. You identify what needs to happen, build a plan, and execute.
• 5+ years in product management, product ownership, or a cross-functional technology leadership role. You’ve shipped real products, not just managed backlogs.
• Deep, hands-on understanding of AI and large language models — not theoretical, but practical. You’ve built or deployed AI-powered workflows, evaluated agent frameworks, or integrated LLM capabilities into production systems.
• Proven ability to manage developers and hold technical teams accountable. You speak engineering well enough to earn credibility and call out scope creep.
• Experience managing website builds and API integrations from kickoff to delivery.
• Working knowledge of HubSpot (CRM, marketing automation, workflows). Experience managing a HubSpot resource or team is a strong plus.
• Excellent communicator across technical and non-technical audiences. You can present to leadership and debug a requirements doc in the same afternoon.
• Comfortable operating with high autonomy, limited supervision, and real accountability for results.
• Experience in real estate, transaction management, or proptech is a plus — but what matters more is your ability to learn a domain fast and drive outcomes within it.

Location: US or Canada
Type: US Applicants - Full-Time; Canadian Applicants - Independent Contractor

About Human Agency

We're scaling rapidly and have a growing pipeline of opportunities that demand exceptional talent across disciplines. Our mission is to bring on individuals, from creative producers to technical experts to entrepreneurial leaders, who can help us realize this next chapter of growth.

We are a company of doers. Leaders roll up their sleeves, teams work flat, and everyone contributes to what ships. Titles don't insulate us from feedback or basics. We invite critique, learn quickly, and keep raising the bar. The best ideas win here, no matter where they come from, because clients trust us to deliver the strongest outcomes every time.

Our clients' missions, products, and bottom lines are sacred. We immerse ourselves in their world, becoming stewards of their goals and partners in solving big problems. Every product, strategy, or asset we create must be both beautiful and functional; practical, usable, and designed for real-world impact.

Humans are our most valuable resource, and we only grow by hiring people who push us forward. Across strategy, engineering, design, data, and operations, we seek out teammates who raise the bar and make us better. Always hire up, never down.

We partner with organizations of all sizes to explore, design, and implement AI strategies that are secure, scalable, and human-centered. We believe AI should amplify human potential, not replace it and we build with that conviction in every engagement. From advisory and tooling to implementation and education, we meet clients where they are at and help them integrate AI in ways that align with their mission and values. Our goal is to empower teams to work smarter, move faster, and unlock new possibilities through thoughtful, responsible innovation.

And through it all, we lead with purpose, love, and adventure. We do meaningful work with people we care about, and we make the ride an adventure worth taking. Because at Human Agency, who we are and how we work are one and the same.

The Opportunity

We are seeking an exceptional Chief of Staff for our Managing Director of AI—someone who combines strategic vision in AI with hands-on execution and technological know-how. You'll be both a force-multiplier and a transformation architect: designing AI-enabled processes, building automated engines that run independently, leading organizational rollouts, and ensuring crisp execution across our AI solutions practice.

This role goes beyond traditional Chief of Staff responsibilities. You'll design processes that leverage AI, build repeatable systems that scale, and lead entire client transformations from discovery to sustained adoption. You have the potential to grow into leading organizational-level AI implementations as the practice expands.

Key Responsibilities

Strategic & Executive Leadership

• Partner with the Managing Director of AI and the executive team to translate priorities into sequenced plans, resourcing, and measurable outcomes
• Lead Human Agency's AI transformation methodology: define offerings, build repeatable frameworks, and establish best practices for AI-enabled workflows
• Design end-to-end processes that embed AI tools (LLMs, automation platforms, knowledge assistants) into client workflows and internal operations
• Translate complex AI capabilities into clear business cases, use cases, and implementation roadmaps for executive audiences
• Produce high-quality executive materials (briefings, roadmaps, decision memos) that drive alignment and decisions
• Identify where AI creates leverage across client engagements and surface strategic opportunities to leadership

AI Process Design & Automated Engines

• Design and build AI-powered systems that operate independently with minimal ongoing intervention—automated engines that scale beyond individual engagements
• Create repeatable processes for AI implementation: discovery frameworks, solution architectures, evaluation routines, and governance models
• Design prompts, playbooks, and knowledge bases optimized for reliability, consistency, and reusability across multiple clients
• Implement lightweight automation using no/low-code tools (Zapier, Make, Airtable) integrated with AI capabilities
• Establish monitoring and feedback loops that enable self-correcting, continuously improving systems
• Document patterns and best practices that can be deployed at scale

Client Service Delivery & Organizational Rollouts

• Own end-to-end delivery on multiple client engagements involving AI-enabled solutions (e.g., knowledge assistants, workflow automations, content summarization, research copilots), from discovery through launch and sustained adoption
• Lead company-wide and client-wide AI adoption initiatives—managing stakeholder alignment, training, governance, and measurement across entire organizations
• Run comprehensive discovery to surface use cases, success metrics, risks, change-management needs, and technical requirements; translate findings into scopes, timelines, and clear acceptance criteria
• Coordinate solution build-outs with internal teammates and external partners: data/connectivity needs, tool selection, integration planning, prompt engineering, and rollout choreography
• Design change-management approaches that address skill gaps, cultural resistance, workflow disruption, and organizational readiness
• Partner with client leadership to sequence rollout phases, manage dependencies, and demonstrate ROI

Quality, Governance & Evaluation

• Set lightweight evaluation routines: accuracy metrics, latency benchmarks, cost tracking, quality assurance processes, and continuous improvement cycles
• Establish prompt/playbook management, basic safety/guardrail checks, and UAT plans
• Build governance frameworks that enable responsible, scalable AI adoption across organizations
• Implement monitoring systems that track performance, usage, and business impact

Stakeholder Management & Enablement

• Lead standing client check-ins, executive presentations, demo readiness, decision logs, and release notes, presenting succinct options and recommendations
• Create comprehensive enablement materials: user guides, FAQs, training programs, and governance playbooks that empower client teams to operate AI systems independently
• Build lasting client relationships through proactive communication, anticipation of needs, and consistent delivery quality
• Present to executive audiences with clarity and confidence, addressing concerns about accuracy, security, and adoption

Operations & Project Management

• Design and continuously improve the operating system for AI work: intake → triage → prioritization → delivery → QA → launch → measurement
• Manage staffing assignments and subcontractor contributions across concurrent workstreams; keep tools current (Asana/Notion, Google Workspace, Slack)
• Budget & scope control: track time against SOWs, flag changes early, manage vendors, and ensure on-budget delivery with clear documentation
• Establish shared dashboards for delivery health (timeline, risk, burn, dependency status) and run tight meeting cadences

Communications & Reporting

• Prepare agendas, briefings, and executive-ready presentations; synthesize complex information into actionable insights
• Provide clear, timely reporting to the Managing Director of AI and stakeholders on progress, risks, and decisions
• Contribute to Human Agency's AI thought leadership through documentation of best practices, case studies, and insights

Who You Are

Experience & Skills

• 5-7+ years in operations, client services, product/program/project management, strategic support, consulting, or transformation roles with direct ownership of client-facing delivery
• Demonstrated success leading multi-stakeholder projects and organizational change initiatives end-to-end (scope, timelines, budgets, risk, and change management)
• Deep AI fluency: hands-on experience implementing LLM-enabled solutions, prompt engineering, workflow automation, and knowledge management systems; can shape prompts/playbooks, evaluate quality, and choose appropriate tools or vendors
• Strong understanding of AI capabilities and limitations, when to use AI, how to design for reliability, and how to manage risks responsibly
• Proven ability to design repeatable processes and systems that scale beyond individual projects
• Experience with no/low-code automation platforms (e.g., Zapier/n8n), knowledge bases, and modern AI tooling
• High proficiency with Google Workspace, Asana/Notion, Slack, and comfort with modern web and content tools; experience preparing polished executive documents and presentations
• Exceptional ability to translate complex technical concepts into clear business value for non-technical audiences
• Bonus: Experience leading organizational-level technology rollouts, background in management consulting, familiarity with responsible AI considerations (privacy, data handling, consent, bias mitigation), or experience in AI/ML product development

Mindset & Traits

• Builder and operator: You design systems that work without constant oversight, not just execute tasks
• Strategic and tactical: Comfortable setting vision while rolling up sleeves for hands-on implementation
• AI pragmatist: Excited about possibilities but grounded in what actually delivers measurable value
• Mission-driven; systems thinker with bias for action
• Clear, persuasive communicator who can influence executive stakeholders and earn trust through delivery
• Calm under pressure with excellent judgment about when to escalate vs. solve independently
• Able to hold strategic context while driving tactical execution across multiple concurrent engagements
• Intellectually curious with strong learning agility, you stay current on AI capabilities and translate findings into practical applications
• Comfortable operating in ambiguity and creating structure where none exists

Considerations

• Education: Bachelor's degree or equivalent experience; an advanced degree in business, technology, or a related field is a plus
• Ethics: Commitment to ethical practices and responsible AI (transparency, human oversight, privacy, bias mitigation)
• Travel: Regular (20–40%) for client activities, organizational rollouts, and events
• Location: Remote-friendly with preference for candidates in major tech hubs

Compensation

This role offers a competitive base salary with performance-based bonuses and potential equity participation. Final compensation will vary based on experience, performance, and location.

Why Join Human Agency

Join a team of thinkers and builders creating meaningful impact across sectors—with autonomy to lead, the resources to succeed, and room to grow. This role offers the opportunity to shape how organizations integrate AI at scale, with clear pathways to increased leadership responsibility as our practice expands.

Equal Opportunity Commitment

Human Agency is an Equal Opportunity Employer. We value diverse backgrounds and strive to build an inclusive culture where everyone feels welcomed and empowered.

Open to remote candidates as well as local candidates in New York, Pittsburgh, or St. Louis

The Role: 

The Patient Services Operations team is at the forefront of our mission to deliver affordable, world-class experiences for every customer. As Senior Manager, Knowledge Management (KM) & Communications, you will own the strategy, architecture, and execution of how information is created, maintained, and delivered across Patient Services — at scale. This is not a traditional KM role. We are looking for a forward-thinking leader who treats knowledge infrastructure as a product, with Artificial Intelligence (AI) and automation at its core. You will drive the evolution of how frontline teams access and apply information — moving from static documentation toward intelligent, dynamic knowledge systems that surface the right information at the right moment. In a highly regulated healthcare environment, you will ensure teams are equipped with clear, compliant, and current knowledge while leading internal communications that drive change, alignment, and continuous improvement.

What You'll Do:

Knowledge Management Strategy & AI Innovation

• Define and own the long-term vision for a modern, AI-enabled knowledge management ecosystem across Patient Services
• Lead the evaluation, implementation, and governance of AI-powered knowledge tools — including Large Language Model (LLM)-assisted content generation, intelligent search, and automated content freshness workflows
• Build a knowledge infrastructure that proactively surfaces relevant information to frontline agents in real time, reducing handle time and improving accuracy
• Partner with Product and Technology teams to integrate knowledge systems into agent-facing tools, chatbots, and customer-facing self-service experiences
• Continuously assess the emerging AI/ML landscape for opportunities to improve content creation, retrieval, personalization, and compliance monitoring

Content Development & Editorial Leadership

• Set and uphold a high editorial bar for all knowledge content — Standard Operating Procedures (SOPs), communications, training materials, and compliance documentation — ensuring content is clear, accurate, empathetic, and user-focused
• Lead a team of content developers and knowledge specialists, setting clear goals, coaching for growth, and building a culture of quality and ownership
• Own the full content lifecycle: creation, review, approval, publication, versioning, and retirement
• Develop scalable frameworks and governance models for content consistency across business units and geographies

Internal Communications

• Oversee all internal communications to Patient Services teams, including operational changes, policy updates, product launches, and leadership messaging
• Design communication strategies that drive comprehension, adoption, and alignment at scale across a distributed frontline workforce
• Measure communication effectiveness and iterate based on data and team feedback

Cross-Functional Execution & Program Management

• Lead knowledge and communications workstreams for new client and product launches — from requirements gathering through go-live and post-launch optimization
• Partner closely with Operations, Compliance, Legal, Training, Product, and Technology to ensure knowledge content meets regulatory standards and operational realities
• Define, track, and report on team Key Performance Indicators (KPIs) — content accuracy rates, time-to-publish, agent utilization, self-service deflection, and others — to drive continuous improvement
• Build scalable mechanisms to ensure global consistency and quality as the business grows

A successful applicant will fit the following criteria:

• Bachelor's degree or equivalent experience
• 7+ years of experience in knowledge management, content strategy, or related fields, with at least 3+ years managing and developing people
• Demonstrated experience leading AI or technology-forward initiatives in a knowledge, content, or operations context — including hands-on work with LLM tools, AI-assisted authoring, or intelligent search platforms
• Proven track record owning KM strategy, governance, and content lifecycle for large-scale frontline or customer service operations
• Strong editorial and written communication skills, with experience leading internal communications through complex operational or policy change
• Experience operating in regulated industries (healthcare, pharmacy, insurance, or similar), with a rigorous focus on accuracy and compliance
• Comfort working with data — defining KPIs, analyzing content performance, and translating insights into actionable improvements
• Experience with enterprise KM platforms, CMS tools, or contact center knowledge bases (e.g., Guru, Confluence, Salesforce Knowledge, or similar)
• A builder's mindset: you see knowledge infrastructure as a product, not a filing system, and you're energized by the opportunity to modernize how teams work

Onsite & Hybrid Work Requirements (Mandatory):

• Open to remote candidates as well as local candidates in New York, Pittsburgh, or St. Louis
• Remote candidates: Regular onsite travel is required. During the first 60 days, travel onsite twice per month for four consecutive days (Mon–Thu). After 60 days, travel onsite once per month for four consecutive days (Mon–Thu)
• Local candidates: Hybrid schedule with a minimum of three (3) days per week in office

Our firm is committed to attracting and retaining professionals who value each other and the service we provide by embracing Teamwork, Collaboration, Client Service, and Innovation. If you are a motivated professional looking for a long-term fit where you can grow in a role, and will be valued and empowered, then we invite you to apply to our Staff Attorney position, in any of our offices.

This position offers a flexible, hybrid working arrangement.

The Position

This is not your average discovery role. We’re looking for a sharp, investigative lawyer who enjoys working as part of a team and thrives on digging deep, spotting critical issues, and navigating complex discovery landscapes with precision. If you enjoy engaging directly and collaborating with in-house and outside counsel and crafting tailored discovery responses that align with both individual case nuances and broader national strategies, this dynamic position is for you. This is not a document review position; this person will be part of a dynamic team that develops and executes national strategies for a large international firm client.

Key Responsibilities:

• Conduct thorough investigations and proactively spot discovery issues unique to each case.
• Collaborate closely with inside counsel and external law firms to address distinct discovery challenges and provide clear, actionable guidance.
• Develop and craft strategic discovery responses tailored to individual cases while ensuring alignment with overarching national litigation strategies.
• Operate in a fast-paced, dynamic legal environment where adaptability and innovative problem-solving are essential.
• Continuously monitor and evaluate discovery processes to identify opportunities for improvement and risk mitigation.

Qualifications

• Juris Doctor (JD) degree from an accredited law school.
• Admission to practice law in relevant jurisdictions.
• Proven experience in litigation discovery, investigations, or related legal roles.
• Strong analytical and issue-spotting skills with attention to detail.
• Excellent communication skills and ability to collaborate effectively with diverse legal teams.
• Excellent writing skills.
• Comfortable working in a dynamic, fast-moving environment.
• Experience with AI large language model is a plus.
• Personal injury litigation background preferred

We are not currently accepting agency submissions for this position.