Staff Site Reliability Engineer

As a Staff SRE at Obsidian, you will define and drive the company-wide reliability vision for a complex, multi-tenant SaaS platform serving enterprise and financial customers. You will operate as a strategic partner to DevOps and Platform Engineering leadership, shaping a unified reliability strategy that scales across the organization.

Your core mandate: ensure Obsidian detects, diagnoses, and communicates system issues before customers are impacted—consistently and predictably.

This is a hands-on technical role that involves architecting and leading the implementation of systems that handle real-world complexity, including upstream SaaS dependencies, sparse and noisy signals, and mission-critical enterprise workloads.

Key Responsibilities

• Reliability Strategy & Architecture - Define and lead long-term reliability strategy across services. Establish end-to-end system visibility frameworks and guide architecture for observability, detection, and resilience.
• Cross-Org Leadership - Partner across teams to embed reliability, standardize SLI/SLOs, and serve as a technical escalation expert.
• Detection & Observability - Build intelligent detection systems (anomaly detection, connector health models) and enable self-service observability.
• Incident Management - Define and evolve a tiered incident communication strategy, improve response practices, and lead postmortems to strengthen reliability and customer trust.
• Execution - Contribute hands-on to system design, monitoring, and debugging across distributed systems and data pipelines.

Required Qualifications

• 5+ years in SRE, Production Engineering, or related roles
• 3+ years operating at a senior or technical leadership level (Staff or equivalent scope)
• Deep expertise in:
• AWS and/or GCP
• Kubernetes and Helm
• Observability stacks (Prometheus, Grafana, or equivalent)
• CI/CD systems (GitLab CI/CD, ArgoCD, etc.)
• Proven experience designing and scaling reliability systems for multi-tenant SaaS platforms
• Strong debugging and systems thinking across distributed microservices and legacy systems
• Demonstrated ability to lead initiatives that improve incident detection, response, and system resilience
• Hands-on engineering approach with a track record of building—not just configuring—reliability systems

Preferred Qualifications

• Experience in B2B SaaS serving enterprise or financial customers
• Familiarity with third-party SaaS connector architectures and ingestion patterns
• Experience building anomaly detection or intelligent alerting systems
• Experience designing customer-facing status pages and incident communication frameworks

Why This Role

• Drive org-wide reliability strategy
• Own and build new detection & observability systems
• Tackle complex distributed systems challenges
• Safeguard critical infrastructure for financial customers

What Success Looks Like

• Issues caught and resolved before customer impact
• Reliability is measurable and continuously improving
• Teams self-serve observability with scalable tools
• Clear, proactive incident communication builds trust
• Reliability becomes a competitive advantage

We're looking for a Platform Engineer to join our Application Platform team. You'll help build and operate the deployment platform, tooling, and developer experience that every engineering team at dunnhumby relies on to ship software to production.

This role can be based from either our London or Manchester office.

You'll join at a moment when we're moving our retail analytics platform from monthly bundled releases to continuous per-service deployment on Kubernetes. You'll be hands-on with that migration in your first few months and then stay on long-term as a core member of the team that owns and evolves the platform. It's a clear path into senior platform engineering for someone who has shipped production Kubernetes work and wants to grow their craft on a platform used by hundreds of engineers.

Responsibilities

• Build and maintain the platform primitives that enable service teams to deploy independently, including ArgoCD application manifests, GitOps repository lifecycles, reusable Helm chart templates, and External Secrets Operator configuration.
• Pair with engineers across product and data teams as they onboard onto the new platform, addressing repeated questions at the template layer to prevent recurring issues.
• Contribute to platform design discussions and help shape the paved road that engineers across dunnhumby build on.
• Author and maintain reusable GitLab CI pipeline templates covering build, test, publish, deploy, and post‑deploy stages.
• Work with service teams to migrate pipelines onto the platform standard and retire bespoke per‑service deployment scripts.
• Continuously improve build times, pipeline reliability, and developer feedback loops.
• Respond to platform alerts during working hours, triaging ArgoCD sync failures, secret synchronisation errors, and flaky pipelines.
• Join the Application Platform on‑call rotation once ramped up (typically after the first few months).
• Instrument platform tooling with structured logging and dashboards to improve visibility without manual investigation.
• Write unit and integration tests to meet SonarQube coverage requirements and create smoke tests for deployment workflows — no green CI, no merge.
• Keep migration playbooks, ArgoCD setup guides, and per‑service pipeline documentation accurate and up to date.
• Share learnings with the squad through regular knowledge‑sharing sessions.

What we expect from you:

• Strong professional software or platform engineering experience, shipping code or infrastructure to production within a team environment
• Hands‑on Kubernetes experience on GKE, AKS, or EKS, with confidence using Deployments, Services, Ingress, ConfigMaps, Secrets, and RBAC
• Helm experience, including writing or substantially modifying charts, with a solid understanding of values files, templates, hooks, and chart dependencies
• Experience with GitLab CI or a comparable CI tool (e.g. GitHub Actions, Azure DevOps, Jenkins, CircleCI), having built or meaningfully modified pipelines rather than just used them
• Production experience with Python or Go, with the ability to discuss real code from professional work
• Confidence working with Linux, Bash, Docker, and standard Unix tooling
• Strong Git practices within a team environment, including branching strategies, PR/MR workflows, review feedback, and merge conflict resolution
• A proven track record of shipping to production in a professional team setting (not solely prototype or personal projects)
• Exposure to ArgoCD, Flux, or another GitOps tool is a strong plus
• Experience with External Secrets Operator, Vault, or similar secret‑sync patterns is beneficial
• Familiarity with Terraform, multi‑cloud environments (GCP and Azure), observability tooling (e.g. New Relic, Datadog, Prometheus, Grafana), or contract testing with Pact would be valuable

About the Role - as a Senior Engineer at Obsidian, you’ll:

• Build or extend data pipelines that power Obsidian’s Agentic AI and supply chain risk products.
• Maintain, improve, and evolve existing systems to ensure performance, resilience, and scalability
• Design and implement APIs and backend services, including multithreaded applications
• Collaborate with product and engineering teams to support key product themes and ensure delivery of high-impact features
• Apply strong software engineering practices to requirements gathering, system design, and code reviews
• Contribute to a fast-moving, collaborative environment where adaptability and teamwork are essential.
• Identify and proliferate best practices in data pipeline design between both the UK and US teams.

What’s in It for You

• You will have direct impact on the core product used by enterprises worldwide
• Work alongside a talented, friendly team in a supportive and collaborative culture
• Grow your skills with opportunities to learn new technologies and engineering practices
• Be part of an innovative, fast-paced environment where your contributions are valued
• Enjoy a hybrid working, with supported remote working and great office spaces in Cheltenham and Manchester

Required Skills & Experience

• 5-7 years of experience in a software engineering role
• Proficiency in one or more modern programming languages such as Python, Go, or SQL
• Experience building backend services, APIs, and multithreaded applications
• Familiarity with containerization and orchestration technologies such as Docker and Kubernetes
• Strong knowledge of relational databases (e.g., Postgres)
• Experience collaborating in team environments and adapting to changing requirements
• Understanding of software design principles and engineering best practices
• Experience with cloud platforms (AWS, GCP), object storage (S3), or event/streaming systems (Kafka, Redis)
• Familiarity with Git for version control and deployment tooling, such as GitLab CI/CD
• Experience in influencing design patterns based on best practices.
• Experience assessing alternative data architecture designs to produce accurate and unbiased recommendations.

Desirable Experience

• Experience with system monitoring and observability tools such as Grafana, Prometheus, or similar platforms
• Understanding of quality engineering (QE) practices across development and testing lifecycles
• Exposure to large-scale distributed systems and performance optimisation
• Experience in forming strong stakeholder relationships to help promote joint working between teams.
• Experience of working alongside both UK and US development teams.

AI Skills & AI-Native Engineering Expectations: As an AI-forward engineering organization, we expect senior engineers to effectively leverage AI tools and understand foundational AI concepts to enhance development efficiency and build AI-ready systems.

AI Engineering Capabilities:

• Leverage AI tools effectively to improve development efficiency and build AI-ready systems.
• Proficient with AI-powered developer tools; able to critically evaluate and refine AI-generated outputs.
• Strong understanding of core AI/ML concepts (LLMs, embeddings, vector databases, inference, evaluation).
• Experience integrating AI/ML APIs and building AI-ready data infrastructure (e.g., for RAG).
• Ensure data quality, governance, observability, reliability, security, and performance in AI-driven systems.