Research Scientist at Jobgether

, , Brazil -

Full Time

Start Date

Immediate

Expiry Date

05 Oct, 26

Salary

0.0

Posted On

07 Jul, 26

Experience

5 year(s) or above

Remote Job

Yes

Telecommute

Yes

Sponsor Visa

Skills

Machine Learning Evaluation, AI Benchmarking, LLM Evaluation, Code Model Evaluation, Research Methodology, Construct Validity, Multimodal Systems, Agentic AI, Technical Review, Dataset Validation, Analytical Skills, English Communication

Industry

Internet Marketplace Platforms

Description

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Research Scientist based in Brazil. Join a cutting-edge AI research environment where you will help define how frontier language models are evaluated and benchmarked. In this highly technical role, you will design rigorous evaluation methodologies, develop high-quality benchmarks, and collaborate with subject matter experts to assess advanced capabilities across reasoning, coding, multimodal systems, and agentic AI. You will play a key role in ensuring evaluation frameworks are scientifically robust, reproducible, and aligned with the evolving needs of leading AI research organizations. This opportunity is ideal for a researcher who enjoys solving complex methodological challenges and contributing to the advancement of trustworthy AI evaluation. \n Accountabilities Design and develop original evaluation methodologies that accurately measure the capabilities of frontier AI models across reasoning, coding, agentic workflows, tool use, and multimodal tasks. Create comprehensive benchmark packages supported by expert-verified ground truth, rigorous quality control processes, deterministic verification methods, and well-calibrated scoring rubrics. Evaluate benchmark quality by ensuring strong construct validity, appropriate task discrimination, sufficient performance headroom, and minimal benchmark contamination. Recruit, assess, and coordinate subject matter experts to develop, validate, and review high-quality evaluation datasets and benchmark content. Serve as the final technical reviewer for evaluation quality, benchmark correctness, and task complexity before external delivery. Collaborate with AI research organizations to understand evaluation objectives and translate research needs into effective benchmarking strategies. Lead evaluation pilots from initial concept through final delivery, ensuring all outputs meet demanding scientific and technical standards. Continuously refine evaluation methodologies to support emerging capabilities and evolving best practices in frontier AI research. Requirements Demonstrated research experience in machine learning evaluation, benchmarking, AI assessment methodologies, or a closely related field. Proven expertise designing or contributing to AI benchmarks, evaluation research, published datasets, or similar research initiatives relied upon by AI organizations. Strong understanding of large language model evaluation, including benchmarking methodologies, scoring rubrics, pass rates, contamination risks, construct validity, and discriminative task design. Deep knowledge of code model evaluation and technical assessment frameworks for advanced AI systems. Experience working with subject matter experts and maintaining rigorous quality standards across technical research projects. Excellent analytical, research, and problem-solving skills with a strong attention to scientific rigor. Outstanding written and verbal communication skills in English. Ability to work independently in a remote, research-driven environment while collaborating effectively with multidisciplinary teams. Spanish language proficiency is considered a plus. Benefits Fully remote contract position based in Brazil. Opportunity to work on frontier AI research and evaluation projects with global impact. High level of ownership and influence over research methodology and benchmark design. Collaboration with leading AI experts, researchers, and technical specialists. Flexible remote working environment that supports autonomy and innovation. Exposure to cutting-edge advancements in large language models and AI evaluation. \n How Jobgether works: We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team. We appreciate your interest and wish you the best! Why Apply Through Jobgether? Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time. #LI-CL1

Responsibilities

Design and develop original evaluation methodologies and high-quality benchmarks to measure the capabilities of frontier AI models. Coordinate with subject matter experts to ensure scientific robustness and validity across reasoning, coding, and multimodal tasks.