LLM Inference Engineer at Majestic Labs

Los Altos, California, United States -

Full Time

Start Date

Immediate

Expiry Date

12 Sep, 26

Salary

0.0

Posted On

14 Jun, 26

Experience

2 year(s) or above

Remote Job

Yes

Telecommute

Yes

Sponsor Visa

Skills

LLM Inference, vLLM, SGLang, Python, C++, KV Cache Management, Distributed Inference, Tensor Parallelism, Pipeline Parallelism, Performance Profiling, TensorRT-LLM, Transformer Architecture, Speculative Decoding, Multi-modal Pipelines, Continuous Batching, Scheduling

Industry

Software Development

Description

The Role In this high-impact role, you are the bridge between cutting-edge custom silicon and production-grade AI. You will own the end-to-end LLM serving stack on Majestic hardware, architecting everything from serving APIs down to KV cache management, batching, and scheduling. Your primary mission is to port leading frameworks like vLLM and SGLang to our accelerator and optimize them for peak performance. Because our architecture offers memory headroom, you won't just match traditional GPUs; you will shatter their limits on throughput, batch sizes, and context lengths. As you hunt down bottlenecks, your insights will directly steer our future kernel, compiler, and hardware development. What You'll Own The serving stack, end to end — bring up and adapt a modern inference framework (vLLM, SGLang, or similar) to run on Majestic hardware. The runtime hot path — continuous batching, the scheduler, paged KV cache, and prefill/decode disaggregation. Distributed inference at scale — tensor, pipeline, and expert parallelism across accelerators, wired into our collective communication library (CCL). The multi-modal pipeline — image, audio, and video preprocessing, encoder integration, and mixed-modality batching. Inference-time techniques — speculative decoding, prefix caching, and structured decoding. End-to-end performance — profile, benchmark, and hunt down bottlenecks across the full serving path, feeding findings back to the kernel, compiler, and hardware teams. Requirements What We're Looking For 3+ years building or operating production LLM inference and serving systems (5+ preferred). Deep, hands-on work with a modern inference framework vLLM, SGLang, TensorRT-LLM, Fireworks, or similar including its scheduler, paged attention / KV cache, model executor, and backend integration points. Strong Python and C++, with the ability to move fluidly between the two. A real grasp of transformer inference the prefill/decode split, KV cache behavior, and how batching dynamics shape latency and throughput. Distributed inference experience tensor and pipeline parallelism across multiple devices. An instinct for performance you can profile an end-to-end stack and chase a regression from the serving API all the way down to the kernel.

Responsibilities

Architect and own the end-to-end LLM serving stack on custom hardware, focusing on porting and optimizing frameworks like vLLM and SGLang. You will optimize the runtime hot path and distributed inference to maximize throughput and context lengths.