1033- AI Engineer (Evaluation Systems) at GoFasti

, , -

Full Time

Start Date

Immediate

Expiry Date

02 Jun, 26

Salary

5000.0

Posted On

04 Mar, 26

Experience

5 year(s) or above

Remote Job

Yes

Telecommute

Yes

Sponsor Visa

Skills

Python, Docker, PostgreSQL, AWS, OpenAI, Anthropic, LLM APIs, Backend Engineering, ML Engineering, Evaluation Systems, LLM-as-judge, Precision/Recall, Asynchronous Programming, A/B Testing, Data Curation, PII Handling

Industry

IT Services and IT Consulting

Description

We Make Remote Work Remarkable • TopTalent from LatAm Hello! We are GoFasti, a Talent-as-a-Service. GoFasti bridges the gap between world-class developers and designers from LatAm and first-class companies around the globe. We need an English-fluent AI Engineer (Evaluation Systems), based in Latin America, available to work remotely. We are looking for someone with exceptional communication and relationship-building skills, who embraces changes while maintaining strong attention to detail. An interested and proactive person, who's constantly learning and improving their skills. Are you the one we are looking for? Responsibilities: Design a structured, configurable evaluation engine Combine deterministic checks with LLM-as-judge verdicts Build calibration workflows using expert-labeled examples Measure precision and recall properly (not raw agreement) Handle delayed outcomes and low-confidence review flows Store structured verdicts that power dashboards and analytics Monitor drift and trigger recalibration when alignment drops Specialized judges for specific failure patterns Content-level evaluation (with careful PII handling) Error analysis pipelines that drive workflow redesign A/B testing for judge versions Eval-driven fine-tuning data curation Requirements: 4+ years total backend / ML engineering experience 2+ years building production AI/LLM systems Experience with Python, Docker and PostgreSQL. Experience with AWS, OpenAI, Anthropic, and other LLM APIs Proven experience building LLM-based systems in production environments where output quality and reliability were critical. Experience developing evaluation, QA, or scoring pipelines to assess model performance. Strong understanding of precision/recall trade-offs and working with real-world data. Ability to design and implement systems that produce reliable, structured outputs from LLMs. Strong Python programming skills, including experience with asynchronous programming. Experience deploying and managing services on AWS. It´s a Plus: Human-in-the-loop workflow design OpenTelemetry familiarity Compensation: The Salary range offered for this position varies from (USD) $4,500 - $5,000 per month, depending on seniority and skillset. This position is for an independent contractor, through a payroll platform. The talent will work REMOTELY allocated at our client. Here are the steps for this process: Application review/approval > Screening interview with GoFasti's team > We build and send your profile to our client > Profile review/approval by client > Interview with the client > Client Assessment > Hiring and onboarding. Once you apply for the job, our team will review your resume. If it meets the requirements, we will contact you and move forward in the process. Note for Candidates Approached Directly: If you were contacted directly by a member of our team and are interested in this opportunity, please do not apply through this link. Instead, reach out to the person who contacted you to coordinate a meeting. Thank you!

Responsibilities

The engineer will be responsible for designing a configurable evaluation engine that combines deterministic checks with LLM-as-judge verdicts, building calibration workflows, and measuring precision and recall accurately. Key tasks also include handling delayed outcomes, storing structured verdicts for analytics, monitoring drift, and developing error analysis pipelines.