Principal Machine Learning Engineer (AI Agents) at Palo Alto Networks

Santa Clara, California, United States -

Full Time

Start Date

Immediate

Expiry Date

01 Jan, 26

Salary

215000.0

Posted On

03 Oct, 25

Experience

10 year(s) or above

Remote Job

Yes

Telecommute

Yes

Sponsor Visa

Skills

Machine Learning, AI, Python, SQL, NLP, Microservices, GCP, RAG Systems, Model Evaluation, CI/CD, Data Analytics, Function Calling, Tool Execution, Context Management, Scalability, Observability

Industry

Computer and Network Security

Description

Company Description Our Mission At Palo Alto Networks® everything starts and ends with our mission: Being the cybersecurity partner of choice, protecting our digital way of life. Our vision is a world where each day is safer and more secure than the one before. We are a company built on the foundation of challenging and disrupting the way things are done, and we’re looking for innovators who are as committed to shaping the future of cybersecurity as we are. Who We Are We take our mission of protecting the digital way of life seriously. We are relentless in protecting our customers and we believe that the unique ideas of every member of our team contributes to our collective success. Our values were crowdsourced by employees and are brought to life through each of us everyday - from disruptive innovation and collaboration, to execution. From showing up for each other with integrity to creating an environment where we all feel included. As a member of our team, you will be shaping the future of cybersecurity. We work fast, value ongoing learning, and we respect each employee as a unique individual. Knowing we all have different needs, our development and personal wellbeing programs are designed to give you choice in how you are supported. This includes our FLEXBenefits wellbeing spending account with over 1,000 eligible items selected by employees, our mental and financial health resources, and our personalized learning opportunities - just to name a few! At Palo Alto Networks, we believe in the power of collaboration and value in-person interactions. This is why our employees generally work full time from our office with flexibility offered where needed. This setup fosters casual conversations, problem-solving, and trusted relationships. Our goal is to create an environment where we all win with precision. Job Description Your Career We are seeking a Principal Software Engineer with deep expertise in designing, building, and scaling AI-powered platforms. In this role, you will help shape the future of our agentic platform by leveraging advanced machine learning (ML) techniques to tackle complex, large-scale challenges and deliver impactful customer experiences. Lead the design, prototyping, and productionization of AI agent systems that solve complex user and business problems in cybersecurity applications. You’ll own agent architectures end-to-end—from planning and orchestration to evaluation, deployment, and observability—while mentoring engineers and shaping our AI strategy. This role is located at our Santa Clara Headquarters Campus 3 days a week. Your Impact Design & build agentic systems: Architect workflows and POCs using frameworks such as Google ADK and LlamaIndex; implement tool use, function calling, and multi-step planning. Retrieval & reasoning: Develop RAG pipelines (indexing, retrieval, reranking), code-interpreter/tool execution flows, and robust context management. Model evaluation: Define evaluation suites for performance, efficiency, safety, and business alignment; analyze latency, quality, and cost trade-offs. Scale & reliability: Deploy models and agents to production; build scalable ML pipelines for batch and real-time/streaming use cases; implement monitoring and guardrails. Platform & CI/CD: Drive end-to-end delivery with modern CI/CD; automate testing, rollout, and experiment tracking. Collaboration & leadership: Partner with ML engineers, data scientists, and product to deliver roadmaps; mentor teammates and lead technical design reviews. Documentation & communication: Maintain clear specs and decision records; communicate complex concepts to technical and non-technical audiences. Strategy & incubation: Contribute to AI product vision; incubate new AI initiatives and design microservices-based solutions on GCP. Qualifications Your Experience Basic qualifications 8+ years in ML, data/analytics, and software engineering with production experience. Strong coding skills in Python and proficiency with SQL, including performance/scalability optimization. Preferred qualifications End-to-end experience designing and deploying RAG systems (indexing strategy, retrieval optimization, reranking). Expertise with LLMs and fine-tuning techniques (e.g., LoRA/QLoRA), prompt/agent design, and function-calling patterns. Familiarity with Google ADK (agents, long-term knowledge/memory) and LlamaIndex (graph construction, query engines). Strong background in NLP and/or recommender systems; experience with evaluation methods and dataset curation. Experience with microservices on GCP (e.g., GKE/Cloud Run, Pub/Sub, Vertex AI, CloudSQL/BigQuery) and real-time streaming. Ability to work independently and in cross-functional teams with excellent written and verbal communication. Hands-on experience with vector search and RAG frameworks. Proven track record deploying ML systems to production with CI/CD and observability. M.S. or Ph.D. in a technical field (or equivalent practical experience). Additional Information The Team Our engineering team is at the core of our products – connected directly to the mission of preventing cyberattacks. We are constantly innovating – challenging the way we, and the industry, think about cybersecurity. Our engineers don’t shy away from building products to solve problems no one has pursued before. We define the industry, instead of waiting for directions. We need individuals who feel comfortable in ambiguity, excited by the prospect of a challenge, and empowered by the unknown risks facing our everyday lives that are only enabled by a secure digital environment. Compensation Disclosure The compensation offered for this position will depend on qualifications, experience, and work location. For candidates who receive an offer at the posted level, the starting base salary (for non-sales roles) or base salary + commission target (for sales/commissioned roles) is expected to be between $175000 - $215000 /YR The offered compensation may also include restricted stock units and a bonus. A description of our employee benefits may be found here. Our Commitment We’re problem solvers that take risks and challenge cybersecurity’s status quo. It’s simple: we can’t accomplish our mission without diverse teams innovating, together. We are committed to providing reasonable accommodations for all qualified individuals with a disability. If you require assistance or accommodation due to a disability or special need, please contact us at [email protected]. Palo Alto Networks is an equal opportunity employer. We celebrate diversity in our workplace, and all qualified applicants will receive consideration for employment without regard to age, ancestry, color, family or medical care leave, gender identity or expression, genetic information, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran status, race, religion, sex (including pregnancy), sexual orientation, or other legally protected characteristics. All your information will be kept confidential according to EEO guidelines. Is role eligible for Immigration Sponsorship? No. Please note that we will not sponsor applicants for work visas for this position.

Responsibilities

Lead the design, prototyping, and productionization of AI agent systems for cybersecurity applications. Own agent architectures end-to-end, from planning and orchestration to evaluation, deployment, and observability.