Senior Applied Scientist - M365 Copilot at Microsoft

Beijing, Beijing, China -

Full Time

Start Date

Immediate

Expiry Date

29 May, 26

Salary

0.0

Posted On

28 Feb, 26

Experience

5 year(s) or above

Remote Job

Yes

Telecommute

Yes

Sponsor Visa

Skills

Telemetry Analysis, Prompt Efficiency, M365 Copilot, Azure Open AI, Metric Validation, Offline Evaluation, Online Evaluation, Technical Ownership, Cross-Team Collaboration, Measurement Frameworks, Performance Metrics, LLMs, Prompt Engineering, Python, C#, Mentorship

Industry

Software Development

Description

Overview As a Senior Applied Scientist specializing in Prompt Efficiency, you will be responsible for analyzing telemetry via M365 Copilot and Azure Open AI, as well as defining and validating metrics to assess prompt efficiency. You will deliver prompt efficiency metrics in both offline (shadow) and online evaluations. This position demands strong technical ownership, effective cross-team collaboration, and the aptitude to establish and apply measurement frameworks that ensure prompts in each feature are optimized for efficiency. Microsoft’s mission is to empower every person and every organization on the planet to achieve more. As employees we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals. Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond. Responsibilities Design, develop, and validate performance metrics to assess prompt efficiency. Lead the identification of dependencies and prepare documentation for evaluation methodologies and prompt efficiency metrics. Ensure the accuracy and integrity of the Prompt Efficiency Scorecard. Proactively identify new opportunities to enhance prompt efficiency. Collaborate with feature teams, platform engineers, and leadership to establish objectives, monitor progress, and promote adoption of cost-saving initiatives. Provide mentorship to junior engineers while fostering a culture dedicated to experimentation, data-driven measurement, and ongoing improvement. Qualifications Required Qualifications: Bachelor's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 4+ years related experience (e.g., statistics predictive analytics, research) OR Master's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 3+ years related experience (e.g., statistics, predictive analytics, research) OR Doctorate in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 1+ year(s) related experience (e.g., statistics, predictive analytics, research) OR equivalent experience. Proficient in metrics evaluation pertaining to Al products. Strong coding skills in Python, C#, or similar languages. Demonstrated expertise in large language models (LLMs), prompt engineering, or deployment of Al models. Outstanding skills in communication, teamwork, and leadership. Other Requirements: Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include but are not limited to the following specialized security screenings: Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter. Preferred Qualifications: Master's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 6+ years related experience (e.g., statistics, predictive analytics, research) OR Doctorate in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 3+ years related experience (e.g., statistics, predictive analytics, research) OR equivalent experience. 3+ years experience creating publications (e.g., patents, libraries, peer-reviewed academic papers). Experience presenting at conferences or other events in the outside research/industry community as an invited speaker. 3+ years experience conducting research as part of a research program (in academic or industry settings). 1+ year(s) experience developing and deploying live production systems, as part of a product team. 1+ year(s) experience developing and deploying products or systems at multiple points in the product cycle from ideation to shipping. Familiarity with Azure OpenAI, metrics platforms, and experimentation frameworks. Experience integrating metrics into product scorecards and flight review processes. This position will be open for a minimum of 5 days, with applications accepted on an ongoing basis until the position is filled. Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, ancestry, citizenship, color, family or medical care leave, gender identity or expression, genetic information, immigration status, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran or military status, race, ethnicity, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable local laws, regulations and ordinances. If you need assistance with religious accommodations and/or a reasonable accommodation due to a disability during the application process, read more about requesting accommodations.

Responsibilities

The role involves analyzing telemetry from M365 Copilot and Azure Open AI to define and validate metrics for prompt efficiency, delivering these metrics in both offline and online evaluations. Responsibilities include designing, developing, and validating performance metrics, leading documentation efforts, and ensuring the accuracy of the Prompt Efficiency Scorecard.