Recorded Future is seeking a Principal Software Engineer to join our Engineering team, responsible for building and scaling the pipelines that power the Security Intelligence Graph. As a Principal Software Engineer, you will work across application code and infrastructure to collect, transform, and deliver structured threat data: tracking malware infrastructure, phishing campaigns, spam networks, command-and-control systems, and more. You will play a key role in driving architectural improvements and ensuring the platform scales effectively as it continues to grow.

What You'll Do:

• Data Pipeline Ownership: Design, build, and maintain scalable data pipelines that collect, transform, and deliver structured threat intelligence data powering the Security Intelligence Graph.
• Architecture & Scalability: Drive meaningful architectural improvements across data and platform systems to ensure long-term scalability, reliability, and performance.
• Full-Stack Engineering: Work across both application code and infrastructure, contributing to backend services, data systems, and platform capabilities.
• Data Processing & Analysis: Apply statistical techniques to large-scale datasets to generate accurate and impactful insights.
• Reliability & Observability: Ensure production-grade pipeline reliability through strong observability, error handling, and failure recovery practices.
• Infrastructure & Deployment: Build and maintain CI/CD pipelines and contribute to infrastructure-as-code practices to support efficient and repeatable deployments.
• Cross-Team Collaboration: Communicate technical concepts clearly and collaborate effectively across teams, geographies, and varying levels of technical expertise.
• Technical Leadership: Engage with technical leaders across teams and contribute to raising the engineering bar through knowledge sharing and best practices.

What You'll Bring:

• At least 5 years of experience in data engineering, backend engineering, or platform engineering.
• Strong proficiency in Python, with experience in Go or Rust considered a plus.
• Experience working with cloud infrastructure, including compute, storage, networking, and IAM fundamentals.
• Hands-on experience with message queues and pub/sub systems such as Kafka or RabbitMQ.
• Experience with SQL databases such as Postgres or AWS RDS, and NoSQL databases such as MongoDB or Elasticsearch.
• Familiarity with data lake storage frameworks such as Delta Lake and graph databases such as AWS Neptune or Neo4j.
• Experience developing REST APIs using frameworks such as Flask, Django, or FastAPI.
• Proven experience architecting and delivering production-grade applications and ETL/ELT pipelines.
• Strong understanding of building reliable data pipelines, including observability, error handling, and failure recovery.
• Experience building and maintaining CI/CD pipelines; GitHub Actions experience is a plus.
• Familiarity with infrastructure-as-code tools; Terraform experience is a plus.
• Comfort working with high-volume, complex, and unstructured data.
• Strong problem-solving skills, including the ability to debug both software and data issues.
• Excellent attention to detail and ability to work independently while delivering high-quality results.
• Interest in or familiarity with security data domains such as threat infrastructure, malware, or network telemetry.
• Bachelor’s and/or Master’s degree in Computer Science, Mathematics, Statistics, Engineering, or equivalent experience.

Preferred Qualifications:

• Experience or interest in cybersecurity data domains such as malware analysis, phishing detection, or network telemetry.
• Experience working with graph-based data systems or security intelligence platforms.

The base salary range for this full-time position is $152,000-$228,500. Our salary ranges are determined by role, level, and location. The salary displayed reflects the range for new hire salaries for the position across all US locations. Within the range, individual pay is determined by state, work location and additional factors, including job-related skills, experience, and relevant education or training. This position may be eligible for incentive compensation, equity, and medical, dental, vision, life insurance and 401K. Your recruiter can share more about the specific details of the compensation and benefit package during the hiring process.

#LI-Remote

In this role, you will lead the development of a comprehensive High Temperature Operating Life (HTOL) Test Software system. Your work will involve designing, implementing, and maintaining a scalable multi-chassis testing platform that performs automated stress and performance testing with real-time monitoring and comprehensive data collection capabilities.

Responsibilities

• System Design & Development: Architect, build, and maintain scalable architecture for a multi-chassis HTOL testing system.
• Orchestration: Develop containerized applications for deployment at scale using Python-based services for chassis coordination and management.
• Hardware Monitoring & Management: Create hardware abstraction layers and develop APIs that represent hardware systems, providing essential capabilities for monitoring and management of those systems.
• Manage Data: Develop data collection pipelines handling sensor data and performance metrics.
• Deploy and Update Software: Create automated deployment and testing pipelines using CI/CD best practices.
• Collaboration with Front-End Teams: Work closely with the frontend team to ensure seamless integration of backend APIs with applications.
• Testing & Documentation: Write automated tests to monitor the reliability and performance of the system; maintain clear and concise documentation for troubleshooting.
• Performance and Reliability: Continuously monitor and optimize performance to reduce response times and improve system scalability; ensure uptime in production environments; establish capacity planning procedures.

Required Skills

• BS and 12+ years of experience or MS and 8+ years of experience; degree in Computer Science, Electrical Engineering, or related field.
• Expert level Python, knowledge of web frameworks such as FastAPI, Flask, Django; strong understanding of API design principles and best practices.
• Experience with containerization and orchestration technologies such as Docker and Docker Compose.
• Experience with one or more databases such as MongoDB, PostgreSQL, Redis, time-series databases.
• Familiarity with testing frameworks such as pytest and integration testing, performance testing tools.
• Experience with CI/CD tools such as GitHub Actions/Runners and Infrastructure as Code tools such as Ansible.
• Experience with hardware integration or embedded systems; interfacing with BMCs, FPGAs, temperature sensors, thermal management, power management systems.

Nice-to-have skills

• Familiarity with real-time data handling and communication protocols, such as gRPC, TCP/IP, WebSockets, message brokers or similar technologies.
• Experience with high-availability, mission-critical systems.
• Experience in the Semiconductor Industry: HTOL, wafer-level testing, burn-in systems, reliability testing.
• Professional Certifications: Agile/Scrum certifications.
• Experience building backend services for web applications like Next.js, proficiency in JavaScript/TypeScript.

About this position:

We’re looking for a Machine Learning Infrastructure Engineer to join our growing Data & Platform team and build the foundation that powers our AI and machine learning capabilities across Later’s product portfolio. As our first dedicated ML Infrastructure Engineer, you will own the systems that support model experimentation, training, deployment, and monitoring at scale.

This role is critical to accelerating our data science initiatives and enabling future AI innovation. You’ll design and operate reliable, secure, and scalable ML infrastructure that empowers data scientists and engineers to ship high-impact models with confidence. If you’re excited about building robust ML systems in a fast-moving environment—and want to define the standard for ML Ops at Later—this is your opportunity.

What you'll be doing:

Strategy

• Define and own the long-term ML infrastructure roadmap, ensuring it supports both current experimentation needs and future AI initiatives.
• Establish best practices for model lifecycle management, deployment standards, monitoring, and governance. 
• Identify infrastructure gaps and proactively design scalable solutions to enable high-velocity ML development.
• Contribute to cross-functional technical planning, ensuring ML systems align with product and platform strategy.

Technical/ Execution

• Design, build, and maintain production-grade model deployment and inference systems using CI/CD pipelines, containerized services (Docker), and API frameworks (e.g., Flask).
• Automate end-to-end ML lifecycle workflows including training pipelines, model validation, registry management, deployment, and rollback strategies.
• Implement robust monitoring systems for model performance, latency, drift detection, and infrastructure health using tools such as CloudWatch, Prometheus, and Grafana.
• Operate across AWS and GCP environments to manage training and inference workloads, including GPU-based infrastructure and BigQuery datasets.
• Develop and maintain infrastructure-as-code (Terraform, CloudFormation) to ensure scalable, repeatable, and secure cloud environments.
• Implement and optimize CI/CD workflows (e.g., GitHub Actions, GitLab CI, Bitbucket Pipelines) for ML and infrastructure automation.

Team / Collaboration

• Partner closely with Data Scientists, Analysts, Platform Engineers, and Product Engineers to support end-to-end ML workflows.
• Translate data science experimentation needs into production-ready infrastructure solutions.
• Serve as the technical bridge between ML experimentation and productized deployment.
• Share knowledge and best practices to elevate ML maturity across teams.

Research/Best Practices

• Stay current on emerging ML Ops practices, tools, and frameworks to continuously improve system reliability and efficiency.
• Evaluate and implement model-serving frameworks (e.g., TorchServe, Seldon, TensorRT) where appropriate.
• Contribute to governance, reproducibility, and auditability standards for ML systems.
• Experiment with new tooling and workflows to improve reproducibility, performance, and developer velocity.

What success looks like:

• ML models move from experimentation to production quickly and reliably, with minimal manual intervention.
• CI/CD pipelines enable safe, repeatable deployments with clear rollback strategies.
• Model performance, drift, and infrastructure health are proactively monitored and observable.
• Infrastructure supports scalable GPU training and real-time inference without bottlenecks.
• Data scientists report improved velocity, reproducibility, and confidence in deploying models.
• ML systems are secure, compliant, and aligned with evolving product and AI strategy.

What you bring:

• 4+ years of experience in ML Ops, ML infrastructure, backend engineering, or related roles supporting production ML systems.
• Experience working in cloud-native environments (AWS and/or GCP) with hands-on deployment of ML workloads.
• Proven track record designing and implementing CI/CD pipelines for ML systems.
• Strong experience with Amazon SageMaker, Docker, Flask-based APIs, and infrastructure automation tools.
• Hands-on experience with ML lifecycle tooling such as MLflow, SageMaker Studio, or Weights & Biases.
• Experience managing container orchestration platforms (Kubernetes, EKS, or GKE).
• Strong programming experience in Python (additional experience in Go, Java, or Scala is a plus).
• Experience working with infrastructure-as-code tools such as Terraform or CloudFormation.
• Familiarity with observability tools such as CloudWatch, Prometheus, Grafana, Datadog, or centralized logging platforms.
• Experience managing GPU-based workloads and scaling training/inference systems.
• Familiarity with data infrastructure tools such as BigQuery and cloud-native data pipelines.
• Bonus: Experience supporting LLMs or generative AI pipelines, distributed training systems, feature stores (e.g., Feast), real-time inference systems, or ML governance frameworks.
• A mindset focused on automation, reliability, performance, and continuous improvement in fast-scaling environments.

How you work: 

• Driven by Impact: You deliver results that matter—prioritizing high-value work, meeting deadlines, and adapting quickly while keeping outcomes clear.
• Strategic & Customer-Centric: You anticipate risks and opportunities, connect decisions to long-term growth, and build trust through proactive insights.
• Curious & Growth-Oriented: You seek knowledge, ask sharp questions, and apply learnings fast—challenging the status quo with a mindset of improvement.
• Collaborative & Resilient: You thrive in change by staying resourceful, solution-focused, and positive—removing roadblocks, sharing insights, and keeping morale high.
• Accountable & Honest: You own your work, hold yourself and others to a high bar, and use transparent feedback to drive growth.
• Emotionally Intelligent: You build trust through empathy and collaboration, foster inclusion, and inspire others with grit, optimism, and integrity.

Our approach to compensation:

We take a market-based & data-driven approach to compensation. We leverage data from trusted third-party compensation sources to help us understand the market value of a role based on function, level, geographic location, and scope. We evaluate compensation bi-annually, including performance and market-related factors.

Our salaries are benchmarked against market Total Cash Compensation for the geographic location of our job posting. Compensation for some roles is structured as On Target Earnings (OTE = base + commission/variable) while for others it is structured as Salary only.

To comply with local legislation and ensure transparency, we share salary ranges on all job postings. Skills, experience and other factors help determine the final salary we offer which may vary from the original range posted. 

Additionally, all permanent team members are eligible to participate in various benefits plans as part of their overall compensation package.

Salary Range: 

$ 145,000 -165,000 

#LI-remote

At Klaviyo, the ARIA (Applications, Resources, Intelligence, and Automations) team is building the AI systems that transform how our employees work. We are looking for a Senior AI Engineer to design, build, and scale production-grade LLM integrations and AI-powered automation tools from the ground up. This role sits at the intersection of applied AI research and full-stack software engineering. You will own everything from LLM integration architecture and RAG system design to CI/CD pipeline management and production observability. If you want to build AI that real people use at scale every day, this is the role for you.

You will report into the Director of AI and App Engineering and ….

How you’ll make a difference:

• Design and build LLM-powered applications: Architect and develop production-grade systems leveraging large language models (GPT-4, Claude, Mistral, Llama) for internal automation, knowledge retrieval, and intelligent workflow assistance.
• Build and maintain AI integration pipelines: Design robust, scalable API integrations between AI models and internal systems such as HRIS, ticketing, CRM, and knowledge bases, ensuring reliability, low latency, and high availability.
• Develop full-stack AI features end-to-end: Own the complete development of AI features across the stack from backend model orchestration and API layers to frontend interfaces that make AI accessible to non-technical employees.
• Architect and manage CI/CD pipelines for AI systems: Build and maintain automated deployment pipelines for AI models and services, including model evaluation frameworks, A/B testing infrastructure, and safe rollout strategies.
• Implement RAG and retrieval systems: Design and build retrieval-augmented generation systems grounded in Klaviyo internal knowledge, ensuring accuracy, relevance, and up-to-date responses at scale.
• Evaluate and fine-tune AI models: Lead benchmarking, evaluation, and fine-tuning of foundation models for Klaviyo use cases, including prompt engineering and parameter-efficient fine-tuning techniques.
• Establish AI engineering best practices: Define coding standards, testing frameworks, and architectural patterns for AI development at Klaviyo, enabling the team to ship reliable AI features consistently.
• Build AI observability and monitoring tooling: Instrument AI systems with latency monitoring, output quality evaluation, hallucination detection, cost dashboards, and production alerting.
• Collaborate with Product and Data Science: Work closely with PMs, data scientists, and ML engineers to translate requirements into technical implementations, maintaining a tight feedback loop between experimentation and production.
• Drive security and compliance in AI systems: Ensure AI integrations follow data privacy best practices, implement guardrails for sensitive data handling, and maintain compliance with Klaviyo security and governance standards.

Who you are: 

• 3+ years of software engineering experience, with at least 2 years focused on building AI, ML, or LLM-powered systems in production.
• Deep expertise with large language models such as OpenAI, Anthropic, Mistral, or Llama, and experience integrating them into production applications via API.
• Strong full-stack engineering skills across Python (FastAPI, Flask, or Django) and TypeScript or JavaScript, with experience in modern frontend frameworks.
• Proven experience designing and building CI/CD pipelines for software or ML systems using tools such as GitHub Actions, Jenkins, or CircleCI.
• Experience with RAG systems, vector databases (Pinecone, Weaviate, pgvector), and embedding models.
• Proficiency with cloud infrastructure (AWS, GCP, or Azure), containerization (Docker, Kubernetes), and infrastructure-as-code tools such as Terraform or Pulumi.
• Experience with prompt engineering, model evaluation frameworks, and LLM observability tools such as LangSmith or Weights and Biases.

We use Covey as part of our hiring and / or promotional process. For jobs or candidates in NYC, certain features may qualify it as an AEDT. As part of the evaluation process we provide Covey with job requirements and candidate submitted applications. We began using Covey Scout for Inbound on April 3, 2025.

Please see the independent bias audit report covering our use of Covey here

At Klaviyo, the ARIA (Applications, Resources, Intelligence, and Automations) team is building the AI systems that transform how our employees work. We are looking for a Senior AI Engineer to design, build, and scale production-grade LLM integrations and AI-powered automation tools from the ground up. This role sits at the intersection of applied AI research and full-stack software engineering. You will own everything from LLM integration architecture and RAG system design to CI/CD pipeline management and production observability. If you want to build AI that real people use at scale every day, this is the role for you.

You will report into the Director of AI and App Engineering and ….

How you’ll make a difference:

• Design and build LLM-powered applications: Architect and develop production-grade systems leveraging large language models (GPT-4, Claude, Mistral, Llama) for internal automation, knowledge retrieval, and intelligent workflow assistance.
• Build and maintain AI integration pipelines: Design robust, scalable API integrations between AI models and internal systems such as HRIS, ticketing, CRM, and knowledge bases, ensuring reliability, low latency, and high availability.
• Develop full-stack AI features end-to-end: Own the complete development of AI features across the stack from backend model orchestration and API layers to frontend interfaces that make AI accessible to non-technical employees.
• Architect and manage CI/CD pipelines for AI systems: Build and maintain automated deployment pipelines for AI models and services, including model evaluation frameworks, A/B testing infrastructure, and safe rollout strategies.
• Implement RAG and retrieval systems: Design and build retrieval-augmented generation systems grounded in Klaviyo internal knowledge, ensuring accuracy, relevance, and up-to-date responses at scale.
• Evaluate and fine-tune AI models: Lead benchmarking, evaluation, and fine-tuning of foundation models for Klaviyo use cases, including prompt engineering and parameter-efficient fine-tuning techniques.
• Establish AI engineering best practices: Define coding standards, testing frameworks, and architectural patterns for AI development at Klaviyo, enabling the team to ship reliable AI features consistently.
• Build AI observability and monitoring tooling: Instrument AI systems with latency monitoring, output quality evaluation, hallucination detection, cost dashboards, and production alerting.
• Collaborate with Product and Data Science: Work closely with PMs, data scientists, and ML engineers to translate requirements into technical implementations, maintaining a tight feedback loop between experimentation and production.
• Drive security and compliance in AI systems: Ensure AI integrations follow data privacy best practices, implement guardrails for sensitive data handling, and maintain compliance with Klaviyo security and governance standards.

Who you are: 

• 3+ years of software engineering experience, with at least 2 years focused on building AI, ML, or LLM-powered systems in production.
• Deep expertise with large language models such as OpenAI, Anthropic, Mistral, or Llama, and experience integrating them into production applications via API.
• Strong full-stack engineering skills across Python (FastAPI, Flask, or Django) and TypeScript or JavaScript, with experience in modern frontend frameworks.
• Proven experience designing and building CI/CD pipelines for software or ML systems using tools such as GitHub Actions, Jenkins, or CircleCI.
• Experience with RAG systems, vector databases (Pinecone, Weaviate, pgvector), and embedding models.
• Proficiency with cloud infrastructure (AWS, GCP, or Azure), containerization (Docker, Kubernetes), and infrastructure-as-code tools such as Terraform or Pulumi.
• Experience with prompt engineering, model evaluation frameworks, and LLM observability tools such as LangSmith or Weights and Biases.

We use Covey as part of our hiring and / or promotional process. For jobs or candidates in NYC, certain features may qualify it as an AEDT. As part of the evaluation process we provide Covey with job requirements and candidate submitted applications. We began using Covey Scout for Inbound on April 3, 2025.

Please see the independent bias audit report covering our use of Covey here