Principal Data or Applied Scientists for Search Relevance Metrics at Microsoft

Redmond, Washington, United States -

Full Time

Start Date

Immediate

Expiry Date

19 Jun, 26

Salary

274800.0

Posted On

21 Mar, 26

Experience

10 year(s) or above

Remote Job

Yes

Telecommute

Yes

Sponsor Visa

Skills

BigData, Data Processing, Hypothesis Testing, Statistical Methods, Machine Learning, Labeling Protocols, LLM-as-a-Judge, Data Annotation, Information Retrieval, Crowdsourcing, Data Pipelines, Experimental Design, A/B Testing, Python, T-SQL, HTML/JS

Industry

Software Development

Description

Overview We are seeking an accomplished Principal Data or Applied Scientists for Search Relevance Metrics to lead the development and evaluation of offline search metrics for Bing. This position is integral to the advancement of search quality measurement, driving data-driven insights through rigorous offline analyses. As a member of the Bing Metrics team, you will leverage state-of-the-art BigData technologies to sample, process, and analyze vast search logs, formulate and validate hypotheses regarding search relevance, and deliver actionable findings to leadership. This role offers the opportunity to collaborate extensively across engineering, product, and research teams, shaping the future of search quality at scale. As Principal Data or Applied Scientists for Search Relevance Metrics , you will directly influence Bing’s search quality and relevance, drive innovation in offline metric development, and enable data-informed decision-making at the highest levels. Your work will shape product strategy and have a broad impact across the Bing organization, fostering collaboration and continuous improvement in search experiences worldwide. Microsoft’s mission is to empower every person and every organization on the planet to achieve more. As employees we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals. Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond. Starting January 26, 2026, Microsoft AI (MAI) employees who live within a 50- mile commute of a designated Microsoft office in the U.S. or 25-mile commute of a non-U.S., country-specific location are expected to work from the office at least four days per week. This expectation is subject to local law and may vary by jurisdiction. Responsibilities Sample large, representative datasets from extensive search log repositories utilizing BigData Map-Reduce frameworks and distributed data platforms. Architect and implement scalable data processing and analysis pipelines for offline metric computation, leveraging modern data engineering best practices. Formulate, test, and validate hypotheses regarding search result quality using advanced statistical methods and machine learning models. Design labeling protocols and manage trained crowd judges and auditors for high-fidelity data annotation and validation of search results. Adapt traditional evaluation workflows to incorporate LLM-as-a-judge and fine-tuned LLMs, ensuring robust and scalable quality assessments. Develop and deliver custom reports, visualizations, and presentations to communicate insights and recommendations to senior leadership. Collaborate across multidisciplinary teams to extend offline metric methodologies and support innovative search experiences. Qualifications Required Qualifications: Doctorate in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 5+ years data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results) OR Master's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 7+ years data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results) OR Bachelor's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 10+ years data science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results) OR equivalent experience. Preferred Qualifications: Doctorate in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 8+ years data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results) OR Master's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 10+ years data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results) OR Bachelor's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 12+ years data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results) OR equivalent experience. Extensive experience in Information Retrieval, including designing, evaluating, and optimizing search and ranking algorithms. Demonstrated ability to develop and validate metrics for Information Retrieval systems, ensuring robust measurement of relevance, accuracy, and user satisfaction. Expertise in crowdscience methodologies, including designing and running large-scale crowdsourcing experiments for data annotation and model evaluation. Proven track record in architecting and deploying large-scale data pipelines for real-time and batch processing of heterogeneous data sources. Solid background in experimental design, statistical analysis, and A/B testing for data-driven product improvements. Ability to lead cross-functional teams and mentor junior scientists in best practices for data science and machine learning. Ability to work independently, solid collaboration and communication skills Familiarity with Python, T-SQL, (HTML/JS for dashboarding). At Microsoft, our mission is to empower every person and organization on the planet to achieve more. We foster a culture of inclusion, growth, and innovation, built on values of respect, integrity, and accountability. If you're passionate about driving meaningful impact, solving complex problems, and contributing to a growing organization, we would love to hear from you. #MicrosoftAI #search# #webxt# #metrics# #informationretrieval# #bigdata# #DS# Data Science IC5 - The typical base pay range for this role across the U.S. is USD $139,900 - $274,800 per year. There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $188,000 - $304,200 per year. Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here: https://careers.microsoft.com/us/en/us-corporate-pay This position will be open for a minimum of 5 days, with applications accepted on an ongoing basis until the position is filled. Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, ancestry, citizenship, color, family or medical care leave, gender identity or expression, genetic information, immigration status, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran or military status, race, ethnicity, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable local laws, regulations and ordinances. If you need assistance with religious accommodations and/or a reasonable accommodation due to a disability during the application process, read more about requesting accommodations.

Responsibilities

This role involves leading the development and evaluation of offline search metrics for Bing by leveraging BigData technologies to analyze search logs and formulate relevance hypotheses. The scientist will architect scalable data pipelines, validate metrics using advanced statistics and ML, and adapt evaluation workflows to incorporate LLMs.