🏖️Agent Evals Specialist - Job Code: 1778280366787 at Coconut

, , Philippines -

Full Time

Start Date

Immediate

Expiry Date

08 Aug, 26

Salary

1450.0

Posted On

10 May, 26

Experience

0 year(s) or above

Remote Job

Yes

Telecommute

Yes

Sponsor Visa

Skills

AI Evaluation, Content Verification, Technical Writing, Quality Assurance, Markdown, English Proficiency, Feedback Writing, Attention To Detail

Industry

Staffing and Recruiting

Description

At Coconut 🥥, we connect founders and businesses with top-tier remote talent while prioritizing meaningful, reliable partnerships. We believe success happens when everyone wins. Clients achieve their goals, and our virtual professionals find purpose and growth in their work. We're currently looking for top talent to fill the role below: Role: Agent Evals Specialist Monthly Rate: $900-1450 Availability: 40 hours/week Time Zone: Preferred overlap anytime during 10AM to 10 PM PST ================================================== To streamline your application, please carefully review and complete ALL THREE items below before proceeding to the application form. What to Prepare: 1️⃣ English Proficiency Rating – You will need to enter your CEFR English proficiency level in the application form. If you’re unsure of your rating, you can take a quick assessment here: EF SET Quick Check. 2️⃣ Coconut VA Profile (Resume) – Before submitting your application, please create your Coconut VA profile using this template: Coconut VA Profile Template. ⚠️⚠️⚠️ YOUR RESUME MUST FOLLOW THIS FORMAT AND SHOULD BE IN ENGLISH. This is a test of how keen you are with details and how well you follow instructions. ⚠️⚠️⚠️ 3️⃣ 1-Minute Introduction Video – Record a short introduction video (preferably using Loom) and provide the link in your application. Before you do, check out these video tips and sample videos. Important: By submitting your video, you grant Coconut permission to share it on our platforms to help increase your chances of being matched with a client. Videos with broken or dummy links will result in disqualification. Experience Needed: 1. Read the source and the AI agent's output side by side. Verify the content was captured accurately. 2. Review what the AI agent did. What it created, changed, or left out. 3. Score a short rubric covering accuracy, coverage, organization, and rule adherence. Full rubric provided at onboarding. 4. Write detailed feedback about the mistake. This is the most important thing you produce since we use it to improve the agent. 5. Submit. Move to the next task. Good to have: • Prior work as an AI trainer, tutor, or evaluator (Outlier, DataAnnotation, xAI, Surge, Mercor, Invisible, Toloka, etc.) • Technical writing, editing, QA, translation, paralegal, or research-assistant background • Markdown familiarity Non-negotiable Requirements: - Strong written English. Can read dense technical content for hours without losing focus - Consistent — your scoring on Monday matches your scoring on Friday - Clear, specific feedback:"section 4 dropped the key requirement from page 17", not "this is confusing" Industry/Tools Experience: - Slack and internal platform We’re all about uplifting lives in Coconut and we make sure we provide benefits that support work-life balance, well-being, and growth. Here’s what you can look forward to as part of the Coconut Community! ✨ ✅ Competitive Salary – We recognize your skills and experience with a great compensation package. ✅ 13th Month Pay – Enjoy an extra month’s salary—because who doesn’t love a little bonus? ✅ Paid Time Off – Recharge with 12 days of PTO to use however you like—vacation, self-care, or just a break from it all! ✅ US Holidays Off – Enjoy 12 paid US holidays throughout the year. ✅ Maternity & Paternity Leave – We support growing families with dedicated leave for new parents. ✅ Comprehensive Healthcare – We’ve got you and your family covered with quality healthcare benefits. ✅ Life Insurance* – Our full-time HMO plan includes life insurance for extra peace of mind. ✅ Mental Health Support – Prioritize your well-being with access to mental health consultations and wellness resources. ✅ Milestone Gifts & Birthday Treats – We celebrate your achievements and make your special day even better! ✅ Exciting Team Experiences – Think island getaways, virtual town halls, and regional meetups—because work should be fun, too! ✅ Stay Connected – Be part of a supportive network where you can learn, grow, and connect with amazing people. Some benefits may vary. Full details will be provided during the onboarding process.

Responsibilities

The specialist will verify AI agent outputs against source materials for accuracy and coverage. They are responsible for scoring outputs based on a rubric and providing detailed feedback to improve agent performance.