Senior MLOps Engineer – LLMOps at TRM Labs

, , United States -

Full Time

Start Date

Immediate

Expiry Date

22 Feb, 26

Salary

230000.0

Posted On

25 Nov, 25

Experience

5 year(s) or above

Remote Job

Yes

Telecommute

Yes

Sponsor Visa

Skills

Python, Containerization, Orchestration, Infrastructure-as-Code, CI/CD, Monitoring, Logging, ML Ops, Model Versioning, Automated Evaluation, Model Serving, LLM Workflows, Cost Monitoring, Latency Optimization, Data Accuracy, Compliance

Industry

Information Services

Description

TRM Labs is a blockchain intelligence company committed to fighting crime and creating a safer world. By leveraging blockchain data, threat intelligence, and advanced analytics, our products empower governments, financial institutions, and crypto businesses to combat illicit activity and global security threats. At TRM, you'll join a mission-driven, fast-paced team made up of experts in law enforcement, data science, engineering, and financial intelligence, tackling complex global challenges daily. Whether analyzing blockchain data, developing cutting-edge tools, or collaborating with global organizations, you'll have the opportunity to make a meaningful and lasting impact. The AI Engineering Team is chartered with enabling next-generation AI applications, with a special focus on Large Language Models (LLMs) and agentic systems. Our mission is to build robust pipelines, high-performance infrastructure, and operational tooling that allow AI systems to be deployed with speed, safety, and scale. We manage petabyte-scale pipelines, serve models with millisecond-level latency, and provide the observability and governance needed to make AI production-ready. We’re also deeply involved in evaluating and integrating cutting-edge tools in the LLM and agent space — including open-source stacks, vector databases, evaluation frameworks, and orchestration tools that unlock TRM’s ability to innovate faster than the market. As a Senior MLOps Engineer with a focus in LLMOps, you’ll be at the core of building and scaling the technical infrastructure for AI/ML systems. You will: Build reusable CI/CD workflows for model training, evaluation, and deployment — integrating Langfuse, GitHub Actions, and experiment tracking, etc. Automate model versioning, approval workflows, and compliance checks across environments. Build out a modular and scalable AI infrastructure stack — including vector databases, feature stores, model registries, and observability tooling. Partner with engineering and data science to embed AI models and agents into real-time applications and workflows. Continuously evaluate and integrate state-of-the-art AI tools (e.g. LangChain, LlamaIndex, vLLM, MLflow, BentoML, etc.). Drive AI reliability and governance, enabling experimentation while ensuring compliance, security, and uptime. Build and enhance AI/ML Model Performance Ensure data accuracy, consistency and reliability, leading to better model training and inferencing Deploy infrastructure to support offline and online evaluation of LLMs and agents — including regression testing, cost monitoring, and human-in-the-loop workflows. Enable researchers to iterate quickly by providing sandboxes, dashboards, and reproducible environments. What We’re Looking For Write high-quality, maintainable software — primarily in Python, but we value engineering ability over language familiarity. Have a strong background in scalable infrastructure, including: Containerization and orchestration (e.g. Docker, Kubernetes) Infrastructure-as-code and deployment (e.g. Terraform, CI/CD pipelines) Monitoring and logging frameworks (e.g. Datadog, Prometheus, OpenTelemetry) Understand and implement ML Ops best practices, including: Model versioning and rollback strategies Automated evaluation and drift detection Scalable model and agent serving infrastructure (e.g. vLLM, Triton, BentoML) Deploy and maintain LLM and agentic workflows in production, including: Monitoring cost, latency, and performance Capturing traces for analysis and debugging Optimizing prompt/response flows with real-time data access Demonstrate strong ownership and pragmatism, balancing infrastructure elegance with iterative delivery and measurable impact. Learn about TRM Speed in this position: Rapid Issue Resolution. TRM Engineers identify and resolve critical onsite issues in minutes to hours, not weeks. We create virtual war rooms, implement fixes, and share lessons with both customer stakeholders and internal teams within 48 hours. Navigating Bureaucracy. We anticipate and address procedural hurdles, build trust with key stakeholders, and find alternative pathways to approvals. This keeps projects moving even in complex environments. Efficient Knowledge Transfer. Engineers document and share updates in real time, ensuring the entire team—onsite and remote—has full visibility into plans, blockers, and resolutions. Knowledge sharing sessions and clear documentation reduce friction and accelerate delivery. About TRM's Engineering Levels: Engineer: Responsible for helping to define project milestones and executing small decision decisions independently with the appropriate tradeoffs between simplicity, readability, and performance. Provides mentorship to junior engineers, and enhances operational excellence through tech debt reduction and knowledge sharing. Senior Engineer: Successfully designs and documents system improvements and features for an OKR/project from the ground up. Consistently delivers efficient and reusable systems, optimizes team throughput with appropriate tradeoffs, mentors team members, and enhances cross-team collaboration through documentation and knowledge sharing. Staff Engineer: Drives scoping and execution of one or more OKRs/projects that impact multiple teams. Partners with stakeholders to set the team vision and technical roadmaps for one or more products. Is a role model and mentor to the entire engineering organization. Ensures system health and quality with operational reviews, testing strategies, and monitoring rigor. The following represents the expected range of compensation for this role: Individual pay is determined by skills, qualifications, experience, and location. The compensation details listed in this posting reflect the US base salary only. The estimated base salary range for this role is $215,000 - $230,000. Additionally, this role may be eligible to participate in TRM’s equity plan. Please note – we factor in the different costs for geographies outside the United States. Life at TRM Labs Leadership Principles Our Leadership Principles shape everything we do—how we make decisions, collaborate, and operate day to day. Impact-Oriented Trailblazer – We put customers first, driving for speed, focus, and adaptability. Master Craftsperson – We prioritize speed, high standards, and distributed ownership. Inspiring Colleague – We value humility, candor, and a one-team mindset. Accelerate your Career At TRM, you’ll do work that matters—disrupting terrorist networks, recovering stolen funds, and protecting people around the world. You will: Work alongside top experts and learn every day. Embrace a growth mindset with development opportunities tailored to your role. Take on high-impact challenges in a fast-paced, collaborative environment. Thrive in a culture of coaching, where feedback is fast, direct, and built to help you level up. What to Expect at TRM TRM moves fast—really fast. We know a lot of startups say that, but we mean it. We operate with urgency, ownership, and high standards. As a result, you’ll be joining a team that’s highly engaged, mission-driven, and constantly evolving. To support this intensity, we’re also intentional about rest and recharge. We offer generous benefits, including PTO, Holidays, and Parental Leave for full time employees. That said, TRM may not be the right fit for everyone. If you're optimizing for work life balance, we encourage you to: Ask your interviewers how they personally approach balance within their teams, and Reflect on whether this is the right season in your life to join a high-velocity environment. Be honest with yourself about what energizes you—and what drains you We’re upfront about this because we want every new team member to thrive—not just survive. The Stakes Are Real Our work has direct, real-world impact: Jumping online after hours to support urgent government requests tracing ransomware payments. Delivering actionable insights during terrorist financing investigations. Collaborating across time zones in real time during a major global hack. Building new processes in days, not weeks, to stop criminals before they cash out. Analyzing blockchain data to recover stolen savings and dismantle trafficking networks. Thrive as a Global Team As a remote-first company, TRM Labs is built for global collaboration. We cultivate a strong remote culture through clear communication, thorough documentation, and meaningful relationships. We invest in offsites, regional meetups, virtual coffee chats, and onboarding buddies to foster collaboration. By prioritizing trust and belonging, we harness the strengths of a global team while staying aligned with our mission and values. Join our mission! We’re looking for team members who thrive in fast-paced, high-impact environments and love building from the ground up. TRM is remote-first, with an exceptionally talented global team. If you enjoy solving tough problems and seeing your work make a difference for billions of people, we want you here. Don’t worry if your experience doesn’t perfectly match a job description— we value passion, problem-solving, and unique career paths. If you’re excited about TRM’s mission, we want to hear from you. Recruitment agencies TRM Labs does not accept unsolicited agency resumes. Please do not forward resumes to TRM employees. TRM Labs is not responsible for any fees related to unsolicited resumes and will not pay fees to any third-party agency or company without a signed agreement. Privacy Policy By submitting your application, you are agreeing to allow TRM to process your personal information in accordance with the TRM Privacy Policy Learn More: Company Values | Interviewing | FAQs

Responsibilities

As a Senior MLOps Engineer, you will build and scale the technical infrastructure for AI/ML systems, focusing on CI/CD workflows, model versioning, and compliance checks. You will also partner with engineering and data science teams to embed AI models into real-time applications.