Senior Manager, Human Evaluation, Insights and Science

at  APPLE SERVICES PTE LTD

SUMMARY

Are you passionate about analyzing data patterns for large scale search engines? As part of ASE, you will be a product expert and analyze of a diverse portfolio of Apple Media Product machine learning model (algorithms) evaluations. The team represents the user’s perspective as they vet new features being introduced to AMP. Your day-to-day responsibilities will include analyzing and creating innovative ideas to help launch new features being introduced to Apple Service, and reporting systematic issues to leadership, engineering, and product teams.

KEY QUALIFICATIONS

• 10+ years of experience working with systems and products that are powered by machine-learned models and an understanding of natural language or speech-enabled products
• 10+ years of experience with model evaluation (information retrieval evaluation, recommender system evaluation, etc. )
• 7+ years of direct people management, demonstrated coaching and mentoring capabilities
• Research experience in regards to evaluation methodologies, evaluation metrics (NDCG, MRR, recall, precision, F-score, Precision@k)
• A demonstrated passion for data analysis, data-informed product innovation, building and shaping data culture
• You use good judgment balancing art and science when visually communicating information (e.g. Tableau, Superset, ggplot, D3).
• Capable of driving projects of varying sizes and scopes - some will take months and some weeks — and you will need to know when to dive deep
• Deep understanding of the software development for machine-learned products and services, and how analytics and human evaluation data can help improve those products
• Ability to manage complex relationships across multiple functions and establish strong partnerships. Outstanding communication and presentation skills, written and verbal, to all levels of an organization
• Good product sense, with the ability to translate between product goals, business goals and technical requirements for data analytics
• Self-motivated and proactive, with demonstrated creative and critical thinking capabilities
• Innate capability to, directly and indirectly, lead large teams for success

PREFERRED QUALIFICATIONS

• Advanced degree in Computer Science, Data Science, Engineering, Humanities or related field.
• Proficiency in scripting or programming languages commonly used in data analysis (e.g., Python, R)
• Experience in Large Language Models and Natural Language Processing

DESCRIPTION

• Oversee the development of a portfolio of products run from the Evaluation organization including, but not limited to: instrumentation, offline evaluation, analytics and reporting.
• Develop project plans, monitor and report on progress, monitor staff workload, manage cross-functional dependencies, and track the evaluation organization to team targets. Keep teams focused on the right priorities to meet aggressive deadlines; clearly communicate project progress to leads and executive team.
• Own evaluation design, planning, and prioritization: Oversee all aspects for multiple projects including specifications, scope, requirements, timelines, and resource allocation.
• Partner with the Head of Human Evaluation to drive the development of a product vision for all measurement and evaluation data products in the Apple Media Products team while ensuring our deep commitment to customer privacy.
• Help the product team to deliver, monitor and improve features as we ship them to production: Evaluate and improve the quality of KPIs for key products by analyzing features, and potentially, how they should be instrumented for evaluation success. Empower others to understand and leverage data for decision making.
• Represent our team with other corporate functions (marketing, product security, legal, etc.)
• Ownership of Evaluation Operations, vendors, budget management, and high data quality ownership through management of a manager and team
• Direct supervision of staff, both ICs and Managers: Performance management, career development, education, training, and most importantly, continuous process improvements. Scale the team with top industry talent.

Please refer the Job description for details