THE IDEAL CANDIDATE:

• You know AWS well. You have opinions about the best choice between fancy managed services and do-it-yourself with EC2 and a ball of scripts. Those opinions change based on the circumstances. You know how and when to save money on infrastructure and how and when to save money on human effort.
• You’re deep on model lifecycles. Our MLOps team not only runs the creation of new models but also makes sure that they are deployable, and when deployed that they’re working correctly. We also know that each model will have a time after it is useful, and we think about what data retention and metadata processing look like for a model that is finished. We’re looking for someone to help ensure that our fleet of devices can be as stable and productive as our fleet of servers.
• You think broadly about machine learning. You’ve got interests that range from LLM security to vector database architecture to linear algebra implementations in hardware. When someone describes a cool new trick in AI vision, you want to know how and why it works.
• You are powerfully motivated. You seek out what could be improved and improve it. You prefer to set your own task list over having one spelled out for you. You’ve got experience with Agile development practices and work comfortably with Scrum or Kanban boards.
• You are powerfully lazy. Automation is a requirement for you. Toil in any capacity is heartbreaking, and you’ve got a long list of different kinds of tools and strategies for eliminating it. You know that the “ML” side of “MLOps” brings wide experimentation that enables fast and confident feature development. You know that the “Ops” side brings rapid deployment, observability, and bug-fixing.
• You are powerfully kind. SafelyYou is a company that is doing real good in the world and that is what drives us. You know that you are working as part of a team and that leading with empathy gets better results in every way.
• You’ve seen things. You’ve had 5-7 years implementing MLOps practices at medium to large scale, and 3-5 years of experience managing AWS resources in a professional setting. You have stories about things that went well and poorly in those categories.

YOUR ROLE AT SAFELYYOU

SafelyYou is looking for a Sr. Software Developer - MLOps to help guide and expand their technology platform. MLOps at SafelyYou involves the upkeep and further development of our experimentation, training, and evaluation systems as well as management and control of our data corpus. The right candidate will be able to use operational skills with an architectural mindset to maintain stability and accountability of the SafelyYou products while enabling rapid growth of new machine learning models and services and rapid problem-solving of issues.

RESPONSIBILITIES:

• Security - First and foremost, we’re responsible for the security of some very personal data. You’ll need to know about the right ways to segment data stores from processing pipelines, segment training data from data undergoing inference in the production environment, and how on-call developers can get access to fix problems. To do that you’ll want a good understanding of network segmentation with security groups and VPC networking, cross-account cross-region cross-VPC network transit, and to be able to use a mix of both AWS roles and zero-trust authorization to grant access at only the correct times. Our data needs to be encrypted both in transit and at rest, so we need to understand and deploy the best use of encryption keys and certificate management.
• Research and Development experience - We want our machine learning researchers to be able to iterate quickly and confidently. Different research areas have different needs and different tools will have the best success. We generally track experiments using Weights and Biases, running python code on nVidia hardware atop CUDA. You’ll want to have an opinion on the right ways to run those experiments, gather the data, process it with Spark or Ray and visualize the results. In each case, we’re responsible for helping developers be productive and happy, with low cycle times and high feature throughput.
• Model deployment - Our machine learning models are only useful inasmuch as they are in the hands of our customers. We’re deploying models that watch streams of data coming from our many different sensors and make determinations off of them, but even more so we want to make sure that the models are working the way we expect. You’ll need a good understanding of docker, docker-compose, python, pytorch, and GStreamer to keep deploying new models. You’ll also need to keep our monitoring stack active. We’re currently running Prometheus and Loki, queried by Grafana, and escalated through OpsGenie for that.
• Stability - We keep the technology at SafelyYou running. We’re deep in AWS and have automated our use of it with Terraform. The apps we use to keep ML development going run in a collection of Helm charts deployed in Kubernetes clusters and EC2 instances. We do our data cataloging in Voxel51, keep metadata in a MongoDB database, and manage job execution in Airflow. You should have ideas about managing and accessing vector data as well, in something like Qdrant or pg_vector. All of those need availability and monitoring.
• Data management - We’ve got a large amount of data coming in from our devices and our apps in the cloud that is used to provide a lot of services. That data needs to be stored securely, cataloged, used for machine learning training, used for analytics, and deleted when necessary.