SRE::Network Operations Engineer III

As a Network Operations Engineer III, you will be a senior member of our network operations team, responsible for overseeing the design, implementation, and optimization of our complex and mission-critical network infrastructure. This role requires a high level of expertise in networking technologies, a strategic mindset, and the ability to lead initiatives that enhance the overall performance, security, and scalability of our network.

Key Responsibilities:

Advanced Network Configuration:

• Lead the design and implementation of high-availability, fault-tolerant data center network architectures.
• Develop and implement strategies for the continuous improvement of network infrastructure.
• Develop and review configuration templates which need to be used in bulk provisioning of network devices in a data center environment.

Network Strategy and Planning:

• Contribute to the development of the organization's long-term network strategy.
• Lead capacity planning initiatives, forecasting future network requirements, and recommending scalable solutions.
• Review HLD and LLD diagrams as part of solution design requirements and review.

Emerging Technologies:

• Stay abreast of industry trends and emerging technologies in networking.
• Evaluate and recommend the adoption of new technologies to improve network efficiency and capabilities.

Mentorship and Leadership:

• Provide mentorship and guidance to junior network operations engineers.
• Lead and participate in knowledge-sharing sessions to enhance the team's overall skill set.

Incident Management and Resolution:

• Lead the on-call engineer team and be the L3 escalation point for them for complex incidents and outages.
• Develop and implement incident response procedures and playbooks.

Vendor Management:

• Collaborate with vendors to evaluate and implement new technologies and solutions.
• Manage relationships with key vendors to ensure optimal support and service.

Qualifications:

• Bachelor's degree in Computer Science, Information Technology, or a related field, or equivalent work experience.
• 7+ years of experience in network operations or a related role.
• In-depth knowledge of data center technologies and network routing protocols.
• Expert-level knowledge of routing and switching technologies, and security principles.
• Strong scripting and automation skills, with proficiency in languages like Python, Go, Shell, Ansible, Terraform or similar.
• Proven experience in designing and implementing large-scale, enterprise-level networks.
• Strong analytical and troubleshooting skills.
• Excellent leadership and communication skills.
• Industry certifications such as Cisco CCIE, Juniper JNCIE, or equivalent would be an added advantage.

Preferred Skills:

• Experience in network routing protocols like BGP, OSPF etc.
• Familiarity with data center technologies like VXLAN, EVPN etc.
• Knowledge of firewall technologies like IDS, IPS, WAF, UTM, NAT, IPsec etc.
• Knowledge of network operating systems like Junos, Cisco IOS/XE/XR, Enterprise SONiC, Cumulus Linux etc.
• Knowledge of network automation and scripting languages like Python, Go, Shell etc.
• Expert-level knowledge of configuration management tools like Ansible and Terraform.
• Expertise in complex fundamentals of Linux.
• Experience in monitoring tools and frameworks like Telegraf, Grafana, Prometheus, ELK stack.
• Experience with cloud networking solutions like Microsoft Azure.
• Knowledge of open-source virtualization technologies like KVM.
• Expertise in advanced network troubleshooting and performance optimization.
• Expertise in the implementation and management of load balancing solutions like Nginx, F5 BigIP etc.
• Contributions to open-source networking projects or industry publications.

This Network Operations Engineer III position represents a significant opportunity for a seasoned professional to make a substantial impact on our organization's network infrastructure. Join our team and be a key contributor to the success and innovation of our data center network infrastructure.

SRE - Service Delivery

Team:
The SRE - System Delivery team at PhonePe is a cross-functional, centralized, and horizontal team. We collaborate with other SRE teams to build various infrastructure services and carry out routine production management activities, including infrastructure scaling, database maintenance, and operating system vulnerability rollouts etc.

Job Overview:

As an SRESD Engineer, you will be part of a team responsible for building, managing, and scaling the production systems of PhonePe to ensure uninterrupted services to users across the country. You will primarily work on well-defined tasks under supervision and by following predefined SOPs.

Responsibilities and Duties:

• Build & manage various distributed systems such as Galera, Nginx, Aerospike, ElasticSearch and RMQ.
• Deploy operating system vulnerability fixes to production systems.
• Perform routine database maintenance activities.
• Build and roll out proxy configurations to production systems.
• Understand the criticality of production infrastructure.
• Be willing to work 24/7 shifts and provide on-call support across the SRE Organisation.

Core Technical Skills Required:

Minimum Qualifications

• Linux CLI Tools for System Administration - Proficiency in using command-line tools for managing Linux systems
• Disk & LVM Management - Hands-on experience with disk management and Logical Volume Manager (LVM)
• System Services - Understanding of managing and configuring system services
• Package Management - Understanding of package management tools and processes
• File Permissions - Understanding of Linux file permission systems
• Crontab & Scheduling - Understanding of cron jobs for task scheduling
• DNS Concepts - Understanding of DNS and its configuration
• Using Salt/Ansible for IaC - Understanding of Infrastructure as Code (IaC) using Salt or Ansible
• Database Concepts & Administration - Understanding of Shards, Partitions and Truncations
• Bash / Python Scripting - Proficiency in writing and debugging Bash / Python Scripts

Knowledge of the following will be a plus

• MariaDB or similar Distributed Database Systems
• Docker or similar Containerization technology
• Nginx or similar Proxy technology
• GIT
• Basic Networking Concepts
• Linux-based Firewall
• Basic understanding of HTTP & TLS Security
• Ability to write Python based scripts for various automations

Softskills:

• Good Interpersonal Communication Skills
• Ability to understand & follow SOPs

Qualifications:

• Minimum of 1-3 years of hands-on experience in Linux / Unix System Administration.

Work Location: Bangalore

Note: This role requires the candidate to be available for on-call shifts on a rotational

basis, covering 24 hours a day, 7 days a week.

SRETools - Software Developer 5-7yrs

About PhonePe : Our goal is to make digital payments so easy, safe and universally accepted that people never feel the need to carry cash or cards again. We believe India is at the cusp of a new mobile revolution, which will change the way we manage our money on the go. We see ourselves facilitating this change, through technology and dogged customer centricity.,At PhonePe, we take extra care to make sure you give your best at work, Everyday! And creating the right environment for you is just one of the things we do. We empower people and trust them to do the right thing. Here, you own your work from start to finish, right from day one.Being enthusiastic about tech is a big part of being at PhonePe.,If you like building technology that impacts millions, ideating with some of the best minds in the country and executing on your dreams with purpose and speed, join us!

Objectives of this Role

• Acquire product requirements and designing an optimal solution, by working with stakeholders across domains
• Develop and maintain in-house cloud architecture components, including components that deal with procurement, provisioning, patching, migration and retirement of instances
• Own the implemented solutions end-to-end, right from inception to deployment and monitoring to regular upkeep
• Identify infrastructure wide, company wide problems and provide a scalable solution
• Measure and increase compliance of solutions 
• Bring a DevOps mindset to implementations 

Skills and Qualifications

• 5+ years experience in IT
• Experience with or knowledge of operating systems; current technologies
• Proficient in at least two of these languages: Python/Go/Rust/C. 
• Good in networking fundamentals/systems knowledge, exposure to open source technologies 
• Good in data structures/algorithms
• Awareness of cloud technologies, Mesos, KVM/QEMU, NodeJS/React will be good to have

Job Description:
Are you passionate about maintaining the reliability and performance of one of worlds largest fintech infrastructure? We are seeking a skilled SRE - System Engineer to join our dynamic team. The ideal candidate will have a solid foundation in Linux/Unix administration and networking, along with a keen understanding of open source technologies, tools, and cloud services. Join us to ensure our services remain always-up and always-available.

Key Requirement:
● Proficiency in Linux/Unix administration, coupled with extensive experience in the field.
● Computer networking proficiency and working knowledge.
● Utilize a wide variety of open source technologies and tools, with a strong emphasis on cloud
services.
● Implement best practices and IT operations to maintain an always-up, always-available service.
● Participate actively in the on-call rotation schedule.
● Communicate effectively in English.
● Display a strong background in Linux networking, including the use of IP, iptables, and IPsec.
● Work with MySQL databases effectively. Working experience in Mariadb,Percona will be a plus.
● Understand and apply coding and scripting skills, with a preference for Perl, python or Golang.
● Experience with automation and configuration management, using tools like SaltStack.
● Hands-on experience with private cloud environments.

Desirable Skills:
● Experience with implementing cloud services on Linux using KVM/QEMU.
● Familiarity with container orchestration technology.
● Knowledge of NoSQL databases, especially Aerospike.
● Proficiency in Perl and Golang.
● Understanding of Galera cluster operations.
● Experience with data center related activities.

Summary

We are seeking a highly motivated and experienced Site Reliability Engineer (SRE) with 7 to 12 years of experience to manage, scale, and ensure the high availability of our core infrastructure. This role involves deep expertise in cloud services, automation, monitoring, and complex networking to support a high-volume, mission-critical environment.

Key Responsibilities

• Cloud & Infrastructure: Configure, maintain, and manage services and packages on Ubuntu Virtual Machines in Azure. Design and manage Azure components for log storage, management, alerting, and monitoring.
• Networking & Connectivity: Configure and maintain complex network components including Azure Firewall, Route Tables, Virtual Network Gateways, and Express Route. Establish and manage IPsec and Express Route connectivity with external environments. Manage routing, troubleshooting connectivity issues, and support network component migrations with minimal downtime.
• Automation & IaC: Drive automation for all BAU tasks using Terraform, Saltstack, Ansible, and scripting languages. Write new Terraform code for infrastructure components.
• Database & Data Management: Set up and manage high-availability services like Mysql and Aerospike. Implement database replication across regions, manage migrations, and ensure data sync. Handle backups of databases, logs, and configurations.
• Monitoring & Observability: Implement and manage monitoring (e.g., Prometheus, Victoria Metrics, Riemann) and centralized logging (Loki) solutions, with visualization on Grafana. Troubleshoot performance and system issues at the OS, platform, or application level.
• Security & Compliance: Manage firewalls and integrate platform and VM-level services with the SOC. Collaborate with Infosec teams to evaluate and fix security vulnerabilities.
• Capacity & Performance: Conduct proactive capacity planning. Manage critical infrastructure components like Nginx, HA Proxy, Docker, and RMQ.
• Incident Management & DR: Participate in an on-call rotation. Structure and lead incident response, Root Cause Analysis (RCA), and post-mortem creation. Set up and support planning and execution of DR sites and failovers.

Required Technical Expertise

• Cloud Platform (Microsoft Azure):
• Core Services: Deep, hands-on experience with Microsoft Azure components, including Virtual Machines (Ubuntu/Linux), Azure Storage Accounts, CosmosDB, and Azure Data Explorer (ADX).
• Networking: Expert-level knowledge in configuring and managing complex Azure networking components: Azure Firewall, Azure Route Tables, Virtual Network Gateways, Azure Express Route, and Azure Private DNS. Must be proficient in setting up and troubleshooting routing using protocols like BGP with on-prem DCs and managing network component migrations with minimal downtime.
• Security/Compliance: Experience integrating platform and VM-level services with the Security Operations Center (SOC) and collaborating with Infosec teams on vulnerability evaluation and remediation.
• Operating Systems & Scripting:
• OS: Expert proficiency in Linux environments, specifically Ubuntu/Linux, for system administration, service configuration, and performance troubleshooting at the OS level.
• High-Level Language: Deep expertise in at least one high-level language (Python, Go, or Java) for writing automation, services, and tooling.
• Shell Scripting: Shell scripting (Bash) mastery is essential for day-to-day operational tasks and automation.
• Monitoring, Observability & Logging:
• Monitoring: Extensive experience implementing and maintaining modern monitoring systems such as Prometheus, Victoria Metrics, and Riemann.
• Logging: Proficiency with centralized log management using Loki for log ingestion, enrichment, lifecycle management, and providing a search/view platform.
• Visualization: Expertise in creating and managing dashboards for visualization and alerting using Grafana.
• Configuration Management & IaC (Infrastructure as Code):
• IaC: Mastery of Terraform for writing new component configurations and building automation for BAU (Business As Usual) tasks.
• Configuration Management: Strong experience with configuration management tools like Saltstack (or Ansible) for automated deployment and configuration of services on VMs.
• Databases & Data Stores:
• High-Availability Data Stores: Hands-on experience setting up, managing, and scaling high-availability databases like Mysql and Aerospike.
• Time-Series/Search: Familiarity with Elastic Search and time-series databases like InfluxDB.
• Replication/DR: Expertise in database replication between different regions, managing database migrations, setting up circular replication, and ensuring data sync during system and network issues.
• Core Infrastructure Services:
• Web/Proxy: Expert management of critical infrastructure components like Nginx and HA Proxy, including proxy management, endpoint addition, header configuration, and writing rewrite rules.
• Messaging/Container: Experience with messaging queues like RMQ (RabbitMQ) and containerization technology like Docker.
• Networking Services: Deep knowledge of DNS and other core network protocols.

Essential Soft Skills & Qualifications

• Ownership and Accountability: A proactive approach to identifying and solving infrastructure challenges before they impact service availability.
• Communication: Excellent written and verbal skills for documenting procedures, creating runbooks, and communicating with technical and non-technical stakeholders.
• Mentorship: (For senior roles) Ability to mentor junior engineers and promote SRE best practices across the organization.
• SLO/SLA Management: Experience defining, monitoring, and meeting Service Level Objectives (SLOs) and Service Level Indicators (SLIs) for critical services.
• Toil Reduction: A commitment to measuring and actively reducing operational toil through automation (e.g., using SRE's Toil Reduction framework).
• Cost Optimization: Experience identifying and implementing cloud resource optimization and cost-saving measures within the Azure environment.

Summary

We are seeking a highly motivated and experienced Site Reliability Engineer (SRE) to manage, scale, and ensure the high availability of our core infrastructure. This role involves deep expertise in cloud services, automation, monitoring, and complex networking to support a high-volume, mission-critical environment.

Key Responsibilities

• Cloud & Infrastructure: Configure, maintain, and manage services and packages on Ubuntu Virtual Machines in Azure. Design and manage Azure components for log storage, management, alerting, and monitoring.
• Networking & Connectivity: Configure and maintain complex network components including Azure Firewall, Route Tables, Virtual Network Gateways, and Express Route. Establish and manage IPsec and Express Route connectivity with external environments. Manage routing, troubleshooting connectivity issues, and support network component migrations with minimal downtime.
• Automation & IaC: Drive automation for all BAU tasks using Terraform, Saltstack, Ansible, and scripting languages. Write new Terraform code for infrastructure components.
• Database & Data Management: Set up and manage high-availability services like Mysql and Aerospike. Implement database replication across regions, manage migrations, and ensure data sync. Handle backups of databases, logs, and configurations.
• Monitoring & Observability: Implement and manage monitoring (e.g., Prometheus, Victoria Metrics, Riemann) and centralized logging (Loki) solutions, with visualization on Grafana. Troubleshoot performance and system issues at the OS, platform, or application level.
• Security & Compliance: Manage firewalls and integrate platform and VM-level services with the SOC. Collaborate with Infosec teams to evaluate and fix security vulnerabilities.
• Capacity & Performance: Conduct proactive capacity planning. Manage critical infrastructure components like Nginx, HA Proxy, Docker, and RMQ.
• Incident Management & DR: Participate in an on-call rotation. Structure and lead incident response, Root Cause Analysis (RCA), and post-mortem creation. Set up and support planning and execution of DR sites and failovers.

Required Technical Expertise

• Cloud Platform (Microsoft Azure):
• Core Services: Deep, hands-on experience with Microsoft Azure components, including Virtual Machines (Ubuntu/Linux), Azure Storage Accounts, CosmosDB, and Azure Data Explorer (ADX).
• Networking: Expert-level knowledge in configuring and managing complex Azure networking components: Azure Firewall, Azure Route Tables, Virtual Network Gateways, Azure Express Route, and Azure Private DNS. Must be proficient in setting up and troubleshooting routing using protocols like BGP with on-prem DCs and managing network component migrations with minimal downtime.
• Security/Compliance: Experience integrating platform and VM-level services with the Security Operations Center (SOC) and collaborating with Infosec teams on vulnerability evaluation and remediation.
• Operating Systems & Scripting:
• OS: Expert proficiency in Linux environments, specifically Ubuntu/Linux, for system administration, service configuration, and performance troubleshooting at the OS level.
• High-Level Language: Deep expertise in at least one high-level language (Python, Go, or Java) for writing automation, services, and tooling.
• Shell Scripting: Shell scripting (Bash) mastery is essential for day-to-day operational tasks and automation.
• Monitoring, Observability & Logging:
• Monitoring: Extensive experience implementing and maintaining modern monitoring systems such as Prometheus, Victoria Metrics, and Riemann.
• Logging: Proficiency with centralized log management using Loki for log ingestion, enrichment, lifecycle management, and providing a search/view platform.
• Visualization: Expertise in creating and managing dashboards for visualization and alerting using Grafana.
• Configuration Management & IaC (Infrastructure as Code):
• IaC: Mastery of Terraform for writing new component configurations and building automation for BAU (Business As Usual) tasks.
• Configuration Management: Strong experience with configuration management tools like Saltstack (or Ansible) for automated deployment and configuration of services on VMs.
• Databases & Data Stores:
• High-Availability Data Stores: Hands-on experience setting up, managing, and scaling high-availability databases like Mysql and Aerospike.
• Time-Series/Search: Familiarity with Elastic Search and time-series databases like InfluxDB.
• Replication/DR: Expertise in database replication between different regions, managing database migrations, setting up circular replication, and ensuring data sync during system and network issues.
• Core Infrastructure Services:
• Web/Proxy: Expert management of critical infrastructure components like Nginx and HA Proxy, including proxy management, endpoint addition, header configuration, and writing rewrite rules.
• Messaging/Container: Experience with messaging queues like RMQ (RabbitMQ) and containerization technology like Docker.
• Networking Services: Deep knowledge of DNS and other core network protocols.

Essential Soft Skills & Qualifications

• Ownership and Accountability: A proactive approach to identifying and solving infrastructure challenges before they impact service availability.
• Communication: Excellent written and verbal skills for documenting procedures, creating runbooks, and communicating with technical and non-technical stakeholders.
• Mentorship: (For senior roles) Ability to mentor junior engineers and promote SRE best practices across the organization.
• SLO/SLA Management: Experience defining, monitoring, and meeting Service Level Objectives (SLOs) and Service Level Indicators (SLIs) for critical services.
• Toil Reduction: A commitment to measuring and actively reducing operational toil through automation (e.g., using SRE's Toil Reduction framework).
• Cost Optimization: Experience identifying and implementing cloud resource optimization and cost-saving measures within the Azure environment.

About the Role:

This role is responsible for managing and maintaining complex, distributed big data ecosystems. It ensures the reliability, scalability, and security of large-scale production infrastructure. Key responsibilities include automating processes, optimizing workflows, troubleshooting production issues, and driving system improvements across multiple business verticals.

Roles and Responsibilities:

• Manage, maintain, and support incremental changes to Linux/Unix environments.
• Lead on-call rotations and incident responses, conducting root cause analysis and driving postmortem processes.
• Design and implement automation systems for managing big data infrastructure, including provisioning, scaling, upgrades, and patching clusters.
• Troubleshoot and resolve complex production issues while identifying root causes and implementing mitigating strategies.
• Design and review scalable and reliable system architectures.
• Collaborate with teams to optimize overall system performance.
• Enforce security standards across systems and infrastructure.
• Set technical direction, drive standardization, and operate independently.
• Ensure availability, performance, and scalability of systems and services through proactive monitoring, maintenance, and capacity planning.
• Resolve, analyze, and respond to system outages and disruptions and implement measures to prevent similar incidents from recurring.
• Develop tools and scripts to automate operational processes, reducing manual workload, increasing efficiency and improving system resilience.
• Monitor and optimize system performance and resource usage, identify and address bottlenecks, and implement best practices for performance tuning.
• Collaborate with development teams to integrate best practices for reliability, scalability, and performance into the software development lifecycle.
• Stay informed of industry technology trends and innovations, and actively contribute to the organization's technology communities.
• Develop and enforce SRE best practices and principles.
• Align across functional teams on priorities and deliverables.
• Drive automation to enhance operational efficiency.

Skills Required:

• Over 7 years of experience managing and maintaining distributed big data ecosystems.
• Strong expertise in Linux including IP, Iptables, and IPsec.
• Proficiency in scripting/programming with languages like Perl, Golang, or Python.
• Hands-on experience with the Hadoop stack (HDFS, HBase, Airflow, YARN, Ranger, Kafka, Pinot).
• Familiarity with open-source configuration management and deployment tools such as Puppet, Salt, Chef, or Ansible.
• Solid understanding of networking, open-source technologies, and related tools.
• Excellent communication and collaboration skills.
• DevOps tools: Saltstack, Ansible, docker, Git.
• SRE Logging and monitoring tools: ELK stack, Grafana, Prometheus, opentsdb, Open Telemetry.

Good to Have:

• Experience managing infrastructure on public cloud platforms (AWS, Azure, GCP).
• Experience in designing and reviewing system architectures for scalability and reliability.
• Experience with observability tools to visualize and alert on system performance.

Job Description: 
Are you passionate about maintaining the reliability and performance of one of worlds largest fintech infrastructure? We are seeking a skilled SRE - System Engineer to join our dynamic team. The ideal candidate will have a solid foundation in Linux/Unix administration and networking, along with a keen understanding of open source technologies, tools, and cloud services. Join us to ensure our services remain always-up and always-available. 

Key Requirement: 
● Proficiency in Linux/Unix administration, coupled with extensive experience in the field. 
● Computer networking proficiency and working knowledge.  
● Utilize a wide variety of open source technologies and tools, with a strong emphasis on cloud services. 
● Implement best practices and IT operations to maintain an always-up, always-available service. 
● Participate actively in the on-call rotation schedule. 
● Communicate effectively in English. 
● Display a strong background in Linux networking, including the use of IP, iptables, and IPsec. 
● Work with MySQL databases effectively. Working experience in Mariadb,Percona will be a plus.  
● Understand and apply coding and scripting skills, with a preference for Perl, python or Golang. 
● Experience with automation and configuration management, using tools like SaltStack. 
● Hands-on experience with private cloud environments. 

Desirable Skills: 
● Experience with implementing cloud services on Linux using KVM/QEMU. 
● Familiarity with container orchestration technology.  
● Knowledge of NoSQL databases, especially Aerospike. 
● Proficiency in Perl and Golang. 
● Understanding of Galera cluster operations. 
● Experience with data center related activities.