Senior AI Engineer at Microsoft

Redmond, Washington, United States -

Full Time

Start Date

Immediate

Expiry Date

03 Mar, 26

Salary

234700.0

Posted On

03 Dec, 25

Experience

5 year(s) or above

Remote Job

Yes

Telecommute

Yes

Sponsor Visa

Skills

AI Engineering, Real-Time Intelligence, Eval Pipelines, Telemetry, CI/CD, Cloud Platforms, Microservices, Containers, Architecture Design, ML Systems Evaluation, Communication Skills, Code Reviews, Mentoring, Responsible AI, Programming Languages, Agentic Systems

Industry

Software Development

Description

- Build and evolve the Real-Time Intelligence evaluations platform: implement offline and online eval pipelines, including golden datasets, human review workflows, and LLM-as-judge / auto-raters for agents, anomaly detectors, and decisioning systems. - Instrument agentic solutions for observability by wiring up telemetry, tracing, structured logging, and dashboards so quality, safety, latency, and cost are easy to monitor and debug. - Integrate evals into the development lifecycle by connecting pipelines to CI/CD, canary and A/B experiments, and phased rollouts, making it simple for partner teams to run and interpret evaluations. - Collaborate and mentor across product, research, and engineering teams, sharing best practices on eval design, LLM-as-judge usage, and Responsible AI, and providing code reviews and guidance that raise the bar for the AI features. - Bachelor's Degree in Computer Science or related technical field AND 4+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience. - Master's Degree in Computer Science or related technical field AND 6+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR Bachelor's Degree in Computer Science or related technical field AND 8+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience. - Demonstrated experience shipping LLM-powered features or agentic systems in production (e.g., copilots, assistants, workflow agents, tool-calling systems). - Experience with at least one major cloud platform (Azure, AWS, or GCP) and modern infrastructure practices (microservices, containers, orchestration, CI/CD). - Proven ability to define architectures, make trade-offs, and lead multi-team projects from design through launch. - Background in evaluation of ML/LLM systems: designing test sets, rubrics, human and auto-rater evaluation pipelines. - Excellent communication skills and the ability to explain complex trade-offs to both technical and non-technical stakeholders. - Master's Degree in Computer Science or related technical field AND 6+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR Bachelor's Degree in Computer Science or related technical field AND 8+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience. - Proven ability to define architectures, make trade-offs, and lead multi-team projects from design through launch. - Background in evaluation of ML/LLM systems: designing test sets, rubrics, human and auto-rater evaluation pipelines. - Excellent communication skills and the ability to explain complex trade-offs to both technical and non-technical stakeholders. azdat azuredata Software Engineering IC4 - The typical base pay range for this role across the U.S. is USD $119,800 - $234,700 per year. There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $158,400 - $258,000 per year. Certain roles may be eligible for benefits and other compensation.

Responsibilities

The Senior AI Engineer will build and evolve the Real-Time Intelligence evaluations platform, implementing various evaluation pipelines and workflows. They will also collaborate across teams to integrate evaluations into the development lifecycle and mentor others on best practices.