AI Quality Engineer (f/m/d) at Voiceline

Munich, Bavaria, Germany -

Full Time

Start Date

Immediate

Expiry Date

16 Sep, 26

Salary

0.0

Posted On

18 Jun, 26

Experience

0 year(s) or above

Remote Job

Yes

Telecommute

Yes

Sponsor Visa

Skills

LLM Production, Observability Tooling, Langfuse, LiteLLM, Automated Assessment Pipelines, E2E Testing, Data Analysis, Metric Definition, German Language

Industry

Software Development

Description

Über Uns Do you want to take ownership from day one and actively shape the future in a growing company? VoiceLine is a fast-growing, VC-backed startup based in Munich. We build an AI-powered solution that not only helps field sales and service teams turn conversations into structured data efficiently, but also helps them be smarter on the road. This enables customer information to be documented, analyzed, and seamlessly integrated into existing systems. Together with leading enterprise customers, we are working to make field sales future ready. Deine Aufgaben The Role As our first AI Quality Engineer, you will own the feedback loop that tells us how well our AI is actually performing. That means building the systems to detect, measure, and surface failures, across report quality, assistant behaviour, and pipeline reliability, so the rest of the team can act on them with confidence. What You Will Work On Build automated assessment pipelines that score generated reports for completeness and accuracy, combining output diffs against user submissions, and LLM-based scoring Instrument the pipeline end-to-end: LLM gateways, traces, dashboards, and observability tools that surface regressions in specific stages Analyse assistant conversations to identify and cluster recurring failure modes, missed intent, wrong field extraction, off-track dialogue Build and maintain E2E test harnesses covering the full voice → report flow contribute to feature work when quality systems are stable Dein Profil Solid experience serving LLM-powered products in production Hands-on with observability tooling, Langfuse, LiteLLM, or similar Analytical instinct: you spot patterns in failures and translate them into actionable metrics Self-directed, comfortable defining your own scope in a small, fast-moving team German language skills - reviewing German-language conversations - is a plus Deine Benefits A dynamic environment with flexible working hours and a strong team culture Competitive and fair compensation package, including virtual equity participation Modern office in the heart of Munich and the flexibility to work remotely Access to Wellpass and attractive sports and wellness offerings to support your well-being Support for public transport (Deutschlandticket)

Responsibilities

The AI Quality Engineer will build systems to detect and measure AI failures across report quality and pipeline reliability. This includes creating automated assessment pipelines and instrumenting observability tools to surface regressions.