POSITION SUMMARY

CIQ is seeking a highly experienced Senior or Principal AI Engineer to own and drive AI/ML innovation across our product portfolio. This role sits at the intersection of AI engineering and systems performance - the right candidate brings deep expertise in model inference optimization, training workflows, and production AI deployment, combined with a strong instinct for performance at the systems level.

In this role, you will be the AI engineering standard-bearer at CIQ. You will design and build turnkey AI workload examples - both internal reference pipelines and customer-facing solutions - ensuring that CIQ’s AI story is always compelling, practical, and demonstrably best-in-class. You will integrate deeply with Fuzzball, CIQ’s cloud-native computing platform, running AI workloads end-to-end through it and helping customers do the same.

KEY RESPONSIBILITIES

This role is leveled as Senior or Principal based on qualifications and demonstrated capabilities.

AI Inference Optimization

• Design, implement, and tune inference pipelines for large language models and other AI workloads, targeting maximum throughput and minimum latency.
• Apply state-of-the-art optimization techniques: quantization (INT4/INT8/FP8), model pruning, speculative decoding, continuous batching, and kernel fusion.
• Optimize inference-serving stacks, including vLLM, TensorRT-LLM, ONNX Runtime, and similar frameworks, for production deployment on CIQ’s OS platform.
• Profile and tune GPU/accelerator utilization across the full inference stack, from model weights and memory bandwidth to CUDA kernels and driver overhead.
• Establish inference performance baselines and regression detection across CIQ’s AI-focused solutions.

AI Training Workflows

• Design and optimize distributed training pipelines for large-scale models, including data, model, tensor, and pipeline parallelism strategies.
• Tune training efficiency through mixed-precision training, gradient checkpointing, activation recomputation, and optimizer-level improvements.
• Benchmark training throughput and scaling efficiency across multi-GPU and multi-node configurations on CIQ’s infrastructure.
• Collaborate with infrastructure and performance teams to resolve training bottlenecks at the network (RDMA/InfiniBand), storage, and OS layers.
• Stay current on frontier model architectures and training techniques, including MoE models, RLHF pipelines, and emerging post-training methods.

Turn-Key AI Examples & Reference Workloads

• Build and maintain a library of turn-key AI workload examples that run on CIQ’s platform, covering inference serving, fine-tuning, batch processing, RAG pipelines, and agentic workflows.
• Develop both internal reference pipelines for CI/testing and customer-facing examples designed for immediate productivity on CIQ’s OS and Fuzzball.
• Package workloads using containers to deliver portable, reproducible AI environments across HPC and cloud-native settings.
• Create compelling, well-documented demos and reference architectures that communicate CIQ’s AI capabilities to technical and business audiences alike.
• Partner with product and customer success teams to translate real-world AI use cases into reusable, production-quality examples.

AI Engineering & Tooling

• Build and maintain AI-powered engineering tooling - leveraging LLM-based agents, automated analysis pipelines, and AI-assisted code generation to accelerate the broader engineering organization.
• Champion an AI-first development culture: identify opportunities where AI tooling can reduce toil, surface insights faster, and improve software quality across CIQ’s products.
• Evaluate and integrate emerging AI frameworks, libraries, and hardware as they become relevant to CIQ’s customers and product roadmap.
• Contribute to open-source AI tooling and frameworks where relevant, reinforcing CIQ’s technical reputation in the community.

Fuzzball Integration

• Develop deep expertise in CIQ’s Fuzzball platform, its architecture, scheduling model, and workload execution environment.
• Integrate AI training, inference, and pipeline workloads into Fuzzball-based CI/CD and production pipelines.
• Contribute to Fuzzball’s AI workload story: ensure the platform is a first-class environment for running AI workloads efficiently and at scale.
• Help characterize and improve Fuzzball’s performance for AI-specific access patterns and resource demands.

Cross-Functional Collaboration

• Develop broad familiarity with the full CIQ product portfolio, including Rocky Linux and RLC (and its variants), Fuzzball, Apptainer, and Warewulf, and understand how AI workloads interact with each layer.
• Collaborate closely with the Performance Engineering team to ensure AI workloads benefit from and contribute to CIQ’s systems-level optimization work.
• Partner with product and customer success teams to translate real-world AI pain points into engineering priorities and measurable outcomes.
• Document and communicate findings clearly, from low-level profiling data to executive-level summaries.
• Contribute to technical publications, conference presentations, and thought leadership that reinforces CIQ’s reputation as an AI-forward infrastructure company.

NEEDED TO SUCCEED

Successful candidates will have: 

• Deep, hands-on expertise in LLM inference optimization: including serving frameworks (vLLM, TensorRT-LLM, ONNX Runtime), quantization techniques, and GPU memory management.
• Strong background in distributed AI training, including frameworks such as PyTorch FSDP, DeepSpeed, Megatron-LM, or JAX/XLA.
• Proven experience building production AI pipelines and packaging AI environments for reproducible, portable deployment (containers, Apptainer/Singularity, or equivalent).
• Fluency with GPU/accelerator profiling tools: NVIDIA Nsight, PyTorch Profiler, CUDA performance analysis, and related tooling.
• Familiarity with HPC environments: job schedulers (Slurm, PBS), parallel filesystems, RDMA/InfiniBand, and MPI,  and the intersection of HPC with modern AI workloads.
• Experience integrating AI workloads into CI/CD pipelines and building automated testing and benchmarking frameworks.
• Comfort using and building with LLM-based tools and agentic frameworks to accelerate engineering work.
• Excellent analytical skills and able to form hypotheses, design experiments, and draw actionable conclusions from complex profiling data.
• Strong written and verbal communication skills; able to present findings to both deeply technical audiences and business stakeholders.
• A collaborative, humble, and always-learning mindset, combined with the confidence to champion AI engineering as a first-class concern.

EDUCATION AND EXPERIENCE

• PhD in Computer Science, Machine Learning, Computer Engineering, or a related field strongly preferred; equivalent industry experience considered.
• 10+ years of industry experience in AI/ML engineering, systems software, or a closely related discipline.
• Demonstrated track record of measurable, published, or production-deployed AI performance improvements at scale.
• Experience working in or with open-source AI ecosystems (PyTorch, Triton, ONNX, Hugging Face, etc.) is a strong plus.
• Background with cloud-native, containerized, and/or HPC computing environments preferred.

CIQ is seeking a highly experienced Senior or Principal Performance Engineer to own and drive system-level and application-level performance across our product portfolio. This is an AI-first role, both in methodology and focus area,  and the right candidate will bring deep expertise in operating system internals, kernel and userspace performance, and the performance demands of modern AI workloads, HPC environments, general-purpose computing, and production service workloads.

In this role, you will be the standard-bearer for performance at CIQ. Our performance-focused solutions must always be the fastest in the industry, and you will be responsible for ensuring that remains true. You will be intimately involved with Fuzzball, CIQ's cloud-native computing platform, learning its architecture end-to-end and integrating workloads - both user-facing and CI/testing pipelines - directly through it.

Position Summary

This role is leveled as Senior or Principal based on qualifications and demonstrated capabilities.

Benchmarking & Profiling

• Design, develop, and maintain comprehensive benchmarking frameworks spanning OS, kernel, and application layers.
• Profile workloads across CPU, memory, I/O, network, and accelerator (GPU/NPU) subsystems to identify bottlenecks and optimization opportunities.
• Establish and own performance baselines across CIQ's product and solutions portfolio.
• Leverage AI-assisted tooling and agentic workflows to accelerate profiling, analysis, and root cause identification.

Regression Detection & Resolution

• Build and maintain automated performance regression-detection pipelines integrated into CI/CD workflows using Fuzzball.
• Identify, triage, and resolve regressions across user space, kernel space, and application layers with urgency and rigor.
• Collaborate across engineering teams to root-cause regressions introduced by upstream kernel changes, compiler updates, or library modifications.

Proactive Performance Engineering

• Drive proactive performance improvements - not just reactive fixes - to keep CIQ solutions ahead of the competition across every layer of the stack.
• Own core operating system performance: kernel subsystem tuning (scheduler, memory management, I/O, networking), system call overhead reduction, and user space library and runtime optimizations.
• Identify and implement kernel-level enhancements, including patches, configuration changes, and upstream contributions that yield measurable performance gains for CIQ's customer workloads.
• Optimize for AI inference and training workloads, including LLM serving, model parallelism, and accelerator utilization.
• Tune performance for HPC workloads, including modeling, simulation, and tightly coupled parallel applications (MPI, OpenMP, etc.).
• Optimize general computing and service workloads - web services, databases, messaging systems, and other production software that runs on CIQ's OS platform.
• Work at all levels of the stack: compiler flags, kernel parameters, scheduler tuning, NUMA topology, memory allocation, and application-level algorithmic improvements.

AI-First Approach

• Champion an AI-first engineering philosophy - use AI tools, agents, and automation to accelerate your own productivity and the quality of performance insights.
• Identify and prioritize optimization opportunities that directly impact AI training throughput and inference latency/cost.
• Stay current on state-of-the-art techniques in ML system performance, including quantization, batching strategies, kernel fusion, and hardware-software co-design.

Fuzzball Integration

• Develop deep expertise in CIQ's Fuzzball platform - its architecture, scheduling, and workload execution model.
• Integrate performance benchmarks, regression tests, and user-facing workloads into Fuzzball-based pipelines.
• Contribute to the performance characterization of Fuzzball itself, ensuring the platform adds minimal overhead and scales efficiently.

Cross-Functional Collaboration

• Develop broad familiarity with the full CIQ product portfolio — including Rocky Linux and RLC (and its variants), Fuzzball, Apptainer (formerly Singularity), and Warewulf - understanding how performance considerations span and interconnect across each.
• Collaborate deeply with the engineering teams behind each product line to surface, prioritize, and deliver performance improvements that benefit customers across the entire CIQ ecosystem.
• Partner with product and customer success teams to translate real-world performance pain points into engineering priorities and measurable outcomes.
• Document and communicate findings clearly - from low-level profiling data to executive-level summaries.
• Contribute to technical publications, conference presentations, and thought leadership that reinforces CIQ's reputation for performance excellence.

NEEDED TO SUCCEED

Successful candidates will have:

• A deep, principled understanding of operating system internals -  Linux kernel scheduler, memory subsystem, I/O stack, and networking.
• Proven experience identifying and resolving performance regressions across kernel and user space in production environments.
• Hands-on expertise with profiling and tracing tools: perf, eBPF/bpftrace, Flamegraphs, VTune, Nsight, strace, ftrace, and similar.
• Strong background in AI/ML workload performance - including inference optimization (TensorRT, ONNX, vLLM, or similar), training efficiency, and GPU/accelerator utilization.
• Experience with HPC workloads: MPI, OpenMP, parallel filesystems, RDMA/InfiniBand, and job schedulers (Slurm, PBS, etc.).
• Familiarity with modern AI-first development workflows and comfort using LLM-based tools to accelerate engineering work.
• Experience building automated performance testing and regression detection pipelines in CI/CD environments.
• Excellent analytical skills -  able to form hypotheses, design experiments, and draw actionable conclusions from complex data.
• Strong written and verbal communication skills; able to present findings to both deeply technical audiences and business stakeholders.
• A collaborative, humble, and always-learning mindset -  combined with the confidence to champion performance as a first-class engineering concern.

EDUCATION AND EXPERIENCE

• PhD in Computer Science, Computer Engineering, or a related field strongly preferred; equivalent industry experience considered.
• 15+ years of industry experience in systems performance engineering, OS development, or a closely related discipline.
• Demonstrated track record of measurable, published, or production-deployed performance improvements at scale.
• Experience working in or with open-source ecosystems (Linux kernel contributions, upstream community engagement) is a strong plus.
• Background with cloud-native, containerized, and/or HPC computing environments preferred.