Splunk's Agent Management team is looking for an experienced software engineer to join a team as Senior Staff (Principal) developer. During your work you will own the design, implementation and operation of a brand-new solution that will manage the life cycle of millions of data streaming agents throughout our customer base. The service will allow our customer's critically important security and observability operations to run uninterrupted while performing tasks such as software upgrades, security improvements and health monitoring.As a Principal Engineer, you will operate beyond a single team, setting technical direction for Agent Management and influencing architecture and engineering practices across multiple teams. You will scale your impact by enabling other senior engineers, driving alignment across the organization, and making high-impact technical trade-offs visible and actionable.The principal engineer will and will be primarily responsible for driving product technical design and architecture in a way that should fulfill needs of Splunk customers currently operating on both cloud-base but also on premise deployment. More than this You will provide technical leadership and architectural direction, ensuring teams can deliver high-quality, scalable solutions efficiently. You will ensure that observability is built into the system, Service Level Objectives (SLO) are defined, monitored and met and that engineers are alerted when performance degrades. You will work with senior members of other teams in the Splunk eco-system to ensure their services provide the required functionality and that integration is seamless. You will also collaborate with Product Management to translate business requirements into technical tasks and provide an estimation for implementing them. Success will be measured by your ability to develop and maintain a service that is performant and serviceable while meeting execution goals. Your previous experience in leading critically important, backend-heavy projects is critical. Despite this your work will also include coordinating all aspects of the software development lifecycle and perspective. Ability to teach and raise the knowledge and skill level of junior engineers on the team is another important success measure.Agent Management presents a set of complex technical challenges, including managing the full lifecycle of thousands of agents across highly diverse customer environments. The system must scale to support deployments with 100,000+ agents, handle large-scale configuration changes and upgrades safely, and operate reliably across hybrid cloud and on-premise architectures. It requires careful design of distributed systems, state management, backward-compatible protocols, and rollout mechanisms to ensure reliability, observability, and security at scale, while balancing performance, operability, and long-term maintainability.Within this role you will be also

Requirements: