Research Scientist, Open Source Technical Safeguards at AI Security Institute

London, England, United Kingdom -

Full Time

Start Date

Immediate

Expiry Date

03 Mar, 26

Salary

145000.0

Posted On

03 Dec, 25

Experience

5 year(s) or above

Remote Job

Yes

Telecommute

Yes

Sponsor Visa

Skills

Applied ML, Trust & Safety Tooling, Content Moderation, Security Engineering, Open-Weight Models, Model Hosting Ecosystems, Python, ML Stack, Data Engineering, Systems Skills, Fingerprinting, Detection Approaches, Technical Writing, Communication, Ethical Judgment, Research Collaboration

Industry

Government Administration

Description

About the AI Security Institute The AI Security Institute is the world's largest and best-funded team dedicated to understanding advanced AI risks and translating that knowledge into action. We’re in the heart of the UK government with direct lines to No. 10 (the Prime Minister's office), and we work with frontier developers and governments globally. We’re here because governments are critical for advanced AI going well, and UK AISI is uniquely positioned to mobilise them. With our resources, unique agility and international influence, this is the best place to shape both AI development and government action. Societal Resilience: Societal Resilience is a multidisciplinary team that studies how advanced AI models can impact people and society. We research the prevalence and severity high-impact societal risks caused by frontier AI deployment, and develop mitigations to address these risks. Core research topics include the use of AI for assisting with criminal activities, preventing critical overreliance on insufficiently robust systems, undermining trust in information, jeopardising psychological wellbeing, or for malicious social engineering. We are interested in both immediate and medium-term risks. Why this team matters One emerging risk area we are concerned with is the use of open weight models to drive risks like child sexual abuse material (CSAM) and non-consensual intimate imagery (NCII) generation. AISI has previously published research on methods for making open weight models more robust against malicious tampering. In this role, you’ll join a strongly collaborative technical research team to help design and develop technical safeguards for open weight models that will reduce the risks of CSAM, NCII, and other risk. We do not expect this role to handle this kind of content directly. About the role: This is a research scientist position focused on developing technical safeguards against tampering with open weight model. This role will focus on mitigating AI-generated CSAM and NCII by targeting the real-world supply chain driving harm: open-weight models, adaptation artifacts (LoRAs, guides), and downstream distribution infrastructure (hosting platforms, app stores, operating systems). Our approach prioritises downstream mitigations and actors beyond frontier model developers. This role will build technical tools, protocols, and evidence that platforms and OS/app ecosystems can adopt. This work belongs inside UK government because effective mitigation requires cross-agency coordination (Home Office, DSIT, Ofcom), engagement with regulated platforms under the Online Safety Act, and credible evidence to inform policy trade-offs across innovation, competition, and child protection. This role will synthesise threat intelligence on how AI generated CSAM and NCII are developed, create scalable screening methodologies that platforms can realistically run, and publish best-practice protocols with NGOs to raise the floor across the ecosystem. You’ll work closely with engineers and domain experts across AISI, as well as external research collaborators at Home Office, Internet Watch Foundation, and Ofcom. Researchers on this team have substantial freedom to shape independent research agendas, lead collaborations, and initiate projects that push the frontier of what evaluations can reveal. Example Projects: Publish a Problem Book framing the technical challenges and research directions for preventing CSAM/NCII misuse across model and hosting layers. Develop threat models for how AI generated CSAM and NCII are created and shared. Design and pilot scalable, automated screening methodologies platforms can run pre-publication on uploads (topic-general prototypes that avoid exposure to illegal content). Develop approaches for identifying and tracking known or novel CSAM LoRAs to enable platform blocking at upload. Co-develop best-practice protocols with NGOs (e.g., Thorn/IWF) for hosting, app store, and OS enforcement. This is an individual contributor role with no line management responsibilities. You will report into a senior Research Scientist overseeing our team’s misuse workstream. Impact Your work will raise safety standards across hosting and distribution layers, reduce the availability of CSAM/NCII-generating artifacts (e.g., LoRAs) on major platforms, inform industry protocols and possibly standards, and provide actionable evidence for government decisions Crucially, we do not expect this role to handle NCII or CSAM material. Role Requirements: We’re flexible on the exact profile and expect successful candidates will meet many (but not necessarily all) of the criteria below. Depending on experience, we will consider candidates at either the RS or Senior RS level. Essential At least 3+ years of relevant experience in applied ML, trust & safety tooling, content moderation, security engineering, or adjacent technical fields; we also welcome strong earlier-career applicants (2–3 years) with demonstrated impact in open-source technical work. Deep familiarity with open-weight image/video models (diffusion, LoRA), model hosting ecosystems (e.g., Hugging Face, GitHub), and the limitations of pre-deployment safeguards. Strong methodological rigor and creativity; able to design automated, scalable evaluations and detection methods that generalise and avoid reliance on illegal content. Strong Python and ML stack (PyTorch/JAX), data engineering, and systems skills; experience building pipelines and tooling that run at platform scale. Knowledge of fingerprinting and detection approaches (e.g., perceptual hashing, embedding-based similarity, behavioural signatures), and their privacy and robustness trade-offs. Excellent writing and communication for technical and policy audiences; ability to translate evidence into practical governance guidance. High agency, ethical judgment, and safe-working practices for sensitive topics. Commit to work from our London office in Whitehall for parts of the week, with flexibility for remote work. We’re looking for full-time commitment but are open to part-time arrangements. Preferred Experience collaborating with hosting platforms, app stores, OS vendors, or regulators (e.g., Ofcom) on safety-by-design initiatives. Familiarity with Online Safety Act requirements and platform trust & safety operations; prior work with NGOs such as IWF, Thorn, or STOPNCII.org. Expertise in diffusion models and adaptation techniques (LoRA), model evaluation, and secure tooling for sensitive domains. Experience with privacy-preserving computation, metadata-poor detection, and standardization efforts (RFCs, protocols). Open-source contributions (tools, libraries) and evidence of leading cross-sector technical projects. Example backgrounds Senior trust & safety engineer who built automated content integrity pipelines for a large platform; strong OSS track record; experience with model hosting ecosystems. Applied ML researcher with a PhD/postdoc in computer vision or ML safety; hands-on with diffusion/LoRA; led evaluations and published tooling used by industry. Security/data engineer with 3+ years building scalable detection systems; experience in fingerprinting, hashing, and privacy-preserving methods; collaborated with regulators/NGOs. What we offer: Impact you couldn't have anywhere else Incredibly talented, mission-driven and supportive colleagues Direct influence on how frontier AI is governed and deployed globally Work with the Prime Minister’s AI Advisor and leading AI companies Opportunity to shape the first & best-resourced public-interest research team focused on AI security Resources & access Pre-release access to multiple frontier models and ample compute Extensive operational support so you can focus on research and ship quickly Work with experts across national security, policy, AI research, and adjacent sciences Growth & autonomy If you’re talented and driven, you’ll own important problems early. 5 development days per year, an annual L&D budget, and travel support for conferences and external collaborations. Freedom to pursue research bets without product pressure Opportunities to publish and collaborate externally Life & family Modern central London office (cafes, food court, gym) or option to work in similar government offices in Birmingham, Cardiff, Darlington, Edinburgh, Salford, or Bristol Hybrid working with opportunities for occasional remote work abroad At least 25 days’ annual leave, 8 public holidays, and extra team-wide breaks Generous paid parental leave (36 weeks of UK statutory leave shared between parents + 3 extra paid weeks + option for additional unpaid time) Plus: 27% government-funded pension contribution on top of salary, work from home equipment and dental insurance Annual salary is benchmarked to role scope and relevant experience. Most offers land between £65,000 and £145,000 (base plus technical allowance), with 27% employer pension and other benefits on top (details on the “what we offer” section on our careers page). This role sits outside of the DDaT pay framework given the scope of this role requires in depth technical expertise in frontier AI safety, robustness and advanced AI architectures. The full range of salaries are available below: Level 3 - Total Package £65,000 - £75,000 inclusive of a base salary £35,720 plus additional technical talent allowance of between £29,280 - £39,280 Level 4 - Total Package £85,000 - £95,000 inclusive of a base salary £42,495 plus additional technical talent allowance of between £42,505 - £52,505 Level 5 - Total Package £105,000 - £115,000 inclusive of a base salary £55,805 plus additional technical talent allowance of between £49,195 - £59,195 Level 6 - Total Package £125,000 - £135,000 inclusive of a base salary £68,770 plus additional technical talent allowance of between £56,230 - £66,230 Level 7 - Total Package £145,000 inclusive of a base salary £68,770 plus additional technical talent allowance of £76,230 In accordance with the Civil Service Commission rules, the following list contains all selection criteria for the interview process. The interview process may vary candidate to candidate, however, you should expect a typical process to include some technical proficiency tests, discussions with a cross-section of our team at AISI (including non-technical staff), conversations with your team lead. The process will culminate in a conversation with members of the senior team here at AISI. Candidates should expect to go through some or all of the following stages once an application has been submitted: Initial interview Technical take home test Second interview and review of take home test Third interview Final interview with members of the senior team Additional Information Internal Fraud Database The Internal Fraud function of the Fraud, Error, Debt and Grants Function at the Cabinet Office processes details of civil servants who have been dismissed for committing internal fraud, or who would have been dismissed had they not resigned. The Cabinet Office receives the details from participating government organisations of civil servants who have been dismissed, or who would have been dismissed had they not resigned, for internal fraud. In instances such as this, civil servants are then banned for 5 years from further employment in the civil service. The Cabinet Office then processes this data and discloses a limited dataset back to DLUHC as a participating government organisations. DLUHC then carry out the pre employment checks so as to detect instances where known fraudsters are attempting to reapply for roles in the civil service. In this way, the policy is ensured and the repetition of internal fraud is prevented. For more information please see - Internal Fraud Register. Security Successful candidates must undergo a criminal record check and get baseline personnel security standard (BPSS) clearance before they can be appointed. Additionally, there is a strong preference for eligibility for counter-terrorist check (CTC) clearance. Some roles may require higher levels of clearance, and we will state this by exception in the job advertisement. See our vetting charter here. Nationality requirements We may be able to offer roles to applicant from any nationality or background. As such we encourage you to apply even if you do not meet the standard nationality requirements (opens in a new window). Working for the Civil Service The Civil Service Code (opens in a new window) sets out the standards of behaviour expected of civil servants. We recruit by merit on the basis of fair and open competition, as outlined in the Civil Service Commission's recruitment principles (opens in a new window). The Civil Service embraces diversity and promotes equal opportunities. As such, we run a Disability Confident Scheme (DCS) for candidates with disabilities who meet the minimum selection criteria. The Civil Service also offers a Redeployment Interview Scheme to civil servants who are at risk of redundancy, and who meet the minimum requirements for the advertised vacancy. Diversity and Inclusion The Civil Service is committed to attract, retain and invest in talent wherever it is found. To learn more please see the Civil Service People Plan (opens in a new window) and the Civil Service Diversity and Inclusion Strategy (opens in a new window).

Responsibilities

Develop technical safeguards against tampering with open weight models to mitigate AI-generated CSAM and NCII. Synthesize threat intelligence and create scalable screening methodologies for platforms.