AI/LLM Safety Engineer at Propio

Leawood, Kansas, United States -

Full Time

Start Date

Immediate

Expiry Date

24 Sep, 26

Salary

0.0

Posted On

26 Jun, 26

Experience

5 year(s) or above

Remote Job

Yes

Telecommute

Yes

Sponsor Visa

Skills

AI Safety, LLM Red Teaming, Python, Threat Modeling, Prompt Engineering, Guardrails, Agent Safety, OWASP LLM Top 10, CI/CD, RAG, Reinforcement Learning, Security Engineering, Data Exfiltration Prevention, AI Governance, Software Engineering, MLOps

Industry

Translation and Localization

Description

Description We are seeking an AI/LLM Safety Engineer to join our AI team and take ownership of how safely our models and agents behave in production; with a focus on AI Safety, Trust & Safety, and Responsible AI. You will design the evaluations that catch unsafe behavior, build the guardrails that stop it, and lead the red-teaming that finds the gaps before our users—or attackers—do. Agent safety is the primary focus of this role: you will help ensure that as our systems gain the ability to call tools and take actions, they do so within well-defined, well-tested boundaries. Key Responsibilities: LLM Safety Evaluation & Red Teaming Design and maintain a safety evaluation framework—adversarial prompt sets, scenario-based test suites, and regression suites—so that every model and agent update is validated before it ships. Lead structured red-teaming exercises covering jailbreaks, prompt injection, tool misuse, and data exfiltration; document findings and drive each issue through to remediation and closure. Guardrails & Runtime Controls Build and iterate on guardrail logic, including input/output filtering, tool-boundary constraints, action validation, sensitive-data redaction, and policy prompting. Integrate safety checks into CI/CD and runtime so that unsafe behavior is intercepted before it reaches users. Agent Safety (primary focus of this role) Perform threat modeling for agentic scenarios: tool-call boundaries, sandbox isolation, and least-privilege access, with particular attention to preventing agents from exfiltrating data or executing irreversible actions through chained tool calls. Conduct safety reviews of reinforcement-learning (RL) environments and trajectory data, partnering with environment and agent engineering teams to embed safety constraints directly into the environments themselves. Monitoring & Observability Instrument AI features for safety with structured logging, tracing, and metrics, enabling detection of unsafe patterns and regressions in production. Governance & Collaboration Prepare evidence for governance reviews—test reports, evaluation summaries, and mitigation validation—aligned with internal Responsible AI standards. Collaborate with Product and UX to improve safety interactions (warnings, confirmations, refusal messaging, and feedback collection), and align evaluation goals with the Research and Data teams. Requirements Bachelor's or Master's degree in Computer Science, Software Engineering, Cybersecurity, or a related technical field—or equivalent practical experience. 4+ years building production software, with direct experience working on—or securing—ML/LLM systems. Strong software engineering skills with the ability to write production-grade code (primarily Python), beyond scripting or notebook prototyping. Solid understanding of LLMs and ML: how models work, prompt engineering, and the safety implications of fine-tuning and RAG (e.g., unsafe retrieval, tool misuse, and data exfiltration). A security mindset with demonstrated threat-modeling ability; able to threat-model AI workflows and familiar with the fundamentals of access control, data retention, and incident response. Familiarity with the LLM attack surface—prompt injection, jailbreaks, data poisoning, and supply-chain risk—and working knowledge of the OWASP LLM Top 10. Hands-on experience with at least one of safety evaluation or red teaming, with the ability to walk through a real finding and how it was remediated. Preferred Qualifications Hands-on experience with industry safety tooling such as garak, PyRIT, promptfoo, Giskard, and NeMo Guardrails, and the ability to articulate the trade-offs between them. Visible output in AI safety or security: publications at relevant venues (e.g., the NeurIPS AI Safety Workshop, USENIX Security, or DEF CON AI Village), open-source contributions, or responsible disclosures on frontier models with public write-ups. Familiarity with AI governance and compliance frameworks (NIST AI RMF, ISO/IEC 42001, EU AI Act) and the ability to translate compliance requirements into concrete engineering tasks. Engineering experience with agents, RL environments, and/or tool use. Practical experience with threat-modeling methodologies such as MITRE ATLAS and STRIDE/PASTA. About Propio Propio is on a mission to make communication accessible to everyone. As a leader in real-time interpretation and multilingual language services, we connect people with the information they need across language, culture, and modality. We are committed to building AI-powered tools that enhance interpreter workflows, automate multilingual insights, and scale communication quality across industries.

Responsibilities

Design and maintain safety evaluation frameworks and lead red-teaming exercises to identify vulnerabilities in LLMs and agents. Build runtime guardrails and implement threat modeling to ensure agents operate within secure, well-defined boundaries.