Principal Product Manager at Microsoft

Redmond, Washington, United States -

Full Time

Start Date

Immediate

Expiry Date

08 May, 26

Salary

274800.0

Posted On

07 Feb, 26

Experience

10 year(s) or above

Remote Job

Yes

Telecommute

Yes

Sponsor Visa

Skills

Product Management, Software Development, AI Evaluation Strategy, Cross-Functional Collaboration, Technical Depth, Communication Skills, Market Disruption, Product Metrics Improvement, Framework Development, Evaluation Infrastructure, Intent Detection, Scenario-Level Scoring, Automation, Certification Requirements, Governance, Compliance

Industry

Software Development

Description

Overview We are looking for a deeply technical and forward thinking Principal Product Manager to lead the Evaluation Strategy for Microsoft Agent 365 Tools, which includes MCP, Skills etc. This role sits at the core of our agentic ecosystem, where large scale model orchestration, multi tool trajectories, grounding data, and evaluation rigor converge. You will be responsible for defining how we measure, validate, and continuously improve the performance of 1P and 3P agentic tools across diverse models (OpenAI, Claude, Microsoft models), orchestrators, and enterprise workflows. Your work will directly determine our product quality bars, release gates, certification processes, and how customers, and the industry perceive the reliability of Microsoft’s agentic platform. Microsoft’s mission is to empower every person and every organization on the planet to achieve more. As employees we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals. Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond. Responsibilities Define and own the evaluation strategy for all 1P and 3P Agentic tools like MCP servers, skills etc. including tool invocation success, tool quality, trajectory evaluation, intent detection, and scenario‑level scoring. Develop a unified framework covering offline evals, online evals, AI‑judge‑based evals, and assertion‑based rubric design. Partner with engineering to evolve internal platforms like Agent 365 Evals, Agent Arena, dashboards, CI/CD‑integrated nightly evals, and metrics pipelines. Create grading frameworks, mapping strategies, and ground truth generation mechanisms, including automation for user‑intent derivation. Establish Cross‑Model, Cross‑Orchestrator Eval Infrastructure i.e. ensure agentic tools reliably work across all major LLMs and orchestrators. Design and maintain evaluation suites that capture model regressions, tool invocation drift, and scenario fidelity as products evolve. Drive alignment with internal partners and ISV teams to ensure consistent evaluation approaches, shared pipelines, and consolidated quality dashboards. Define product readiness criteria for 1P/3P tools, aligning certification requirements for partner‑built agentic tools. Partner with responsible AI, security, governance, and compliance teams to ensure eval frameworks respect enterprise boundaries and safety constraints. Track the latest developments in multi‑agent evaluation frameworks, trajectory alignment research, and AI behavioral evals. Bring state‑of‑the‑art thinking from academia, industry, and trajectory evals to shape Microsoft's enterprise agentic tools evaluations strategy. Qualifications Required Qualifications: Bachelor's Degree AND 8+ years experience in product/service/program management or software development OR equivalent experience. Other Requirements: Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include but are not limited to the following specialized security screenings: Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter. Preferred Qualifications: Bachelor's Degree AND 12+ years experience in product/service/program management or software development OR equivalent experience. 4+ years experience taking a product, feature, or experience to market (e.g., design, addressing product market fit, and launch, internal tool/framework). 6+ years experience improving product metrics for a product, feature, or experience in a market (e.g., growing customer base, expanding customer usage, avoiding customer churn). 6+ years experience disrupting a market for a product, feature, or experience (e.g., competitive disruption, taking the place of an established competing product). Demonstrated technical depth across LLMs and line of business systems, with proven experience leading AI/LLM evaluation strategy—including offline/online eval frameworks, rubric and AI judge design, and defining measurable quality bars for agentic tools and orchestration workflows. Cross-functional collaboration skills, with the ability to influence across engineering, research, design and business teams. Exceptional written and verbal communication skills, with a knack for storytelling and clear articulation of complex ideas. High tolerance for ambiguity and a bias for action in dynamic environments. #BICJOBS #AGENT365 Product Management IC5 - The typical base pay range for this role across the U.S. is USD $139,900 - $274,800 per year. There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $188,000 - $304,200 per year. Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here: https://careers.microsoft.com/us/en/us-corporate-pay This position will be open for a minimum of 5 days, with applications accepted on an ongoing basis until the position is filled. Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, ancestry, citizenship, color, family or medical care leave, gender identity or expression, genetic information, immigration status, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran or military status, race, ethnicity, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable local laws, regulations and ordinances. If you need assistance with religious accommodations and/or a reasonable accommodation due to a disability during the application process, read more about requesting accommodations.

Responsibilities

Define and own the evaluation strategy for all 1P and 3P Agentic tools, including tool quality and trajectory evaluation. Partner with engineering to evolve internal platforms and establish evaluation infrastructure across major LLMs and orchestrators.