PhD on Developing a User-centric AI evaluation framework for LLMs

at  TU Eindhoven

JOB DESCRIPTION

Do you want to develop methods to better evaluate, understand and improve human-AI interaction in the context of value-aligned Large Language models? Do you want to conduct impactful user research in close collaboration with scientists and engineers in a EU-funded doctoral network, helping to promote fair and transparent generative AI?
We are seeking a highly motivated doctoral candidate to join our research team as part of a prestigious Marie Skłodowska-Curie Actions (MSCA) doctoral network: AlignAI. The AlignAI doctoral network aims to train doctoral candidates to develop, evaluate, and engage with Large Language Models (LLMs). It focuses on aligning these models with human values to ensure their development and deployment are ethically sound and socially beneficial. By integrating expertise from social sciences, humanities, and technical disciplines, the project will address critical issues such as explainability and fairness, thereby ensuring LLMs contribute positively to education, mental health, and news consumption.
The doctoral candidate in this position will develop and test a user-centric AI evaluation framework that triangulates subjective metrics such as perceived transparency, understandability, fairness and cognitive explainability (mental models), with objective measures of explanation and model quality. Quantitative and qualitative user studies will be developed iteratively throughout the project within the three use cases of the AlignAI consortium: Education, positive health and online news consumption. The candidate will support the validation of AI tools developed by other candidates within alignAI consortium and final validation step with user groups towards the end of the project.
The doctoral candidate will be supervised by associate Professor Martijn Willemsen and embedded within the Human-Technology Interaction group, Department of Industrial Engineering and Innovation Sciences, Eindhoven University of Technology (TU/e). Additional supervision will be provided within the AlignAI doctoral network by Assistant Professor Sneha Das (Computer Science, DTU) and Professor Daniel Gatica- Perez (Digital Humanities,EPFL). The candidate will be affiliated with the Eindhoven Artificial Intelligent Systems Institute (EAISI), and will be expected to contribute actively and positively to Human-AI interaction research communities in Eindhoven, as well as to attend workshops, summer schools, and research visits (secondments) at AlignAI partner institutions as part of the MSCA doctoral network.

JOB REQUIREMENTS

• A master’s degree (or an equivalent university degree) in Human-computer Interaction, Human-Technology Interaction, Computer Science, Cognitive Science, Artificial Intelligence or related disciplines
• Candidate is expected to have a multidisciplinary background that covers user aspects / psychology, research methods and engineering skills in the field of AI
• A research-oriented attitude.
• Applicants must not own a doctoral degree at the date of recruitment.
• Applicants may be of any nationality, but must not have resided or carried out their main activity (work, studies, etc.) in the Netherlands for more than 12 months in the 36 months immediately before the recruitment date.
• Ability to work in an interdisciplinary team and interested in collaborating with industrial partners.
• Motivated to develop your teaching skills and coach students.
• Fluent in spoken and written English (C1 level).

ABOUT US

Eindhoven University of Technology is an internationally top-ranking university in the Netherlands that combines scientific curiosity with a hands-on attitude. Our spirit of collaboration translates into an open culture and a top-five position in collaborating with advanced industries. Fundamental knowledge enables us to design solutions for the highly complex problems of today and tomorrow.

• Develop and test a user-centric AI evaluation framework under supervision of Dr. Martijn Willemsen
• Collaborate with other doctoral candidates in the consortium on validation of the framework by means of several user studies performed within the three use cases
• Engage in secondments at DTU and EPFL as part of the project’s training activities.
• Contribute to academic publications, project reports, and other dissemination activities.
• Participate in structured training programs and network-wide events organized by the consortium.
• Collaborate with international partners and other Doctoral Candidates within the network.
  This position is one of four PhD positions being offered at TU/e through the AlignAI doctoral network, the others being in Philosophy & Ethics (Carlos Zednik) and Industrial Design (Jesse Benjamin & Stephan Wensween).