Senior Data Engineer

at  Streetbees

Streetbees is building the world’s first marketplace for business intelligence, connecting brands and agencies with real people on the ground via our innovative and chat-style app to gather real-time and in-the-moment insights from 4.5 million users worldwide.
We have a very strong engineering team and platform, combining the best in machine learning, data management pipelines, web platforms and mobile apps. We work with Python for our data science, machine learning, data pipelines and data APIs, Ruby on Rails for our web apps, and Swift and Kotlin for our mobile apps. Everything is entirely hosted in AWS, all repos in Github, everything containerised and continuously deployed.
An opportunity exists for a Senior Python Data Engineer to join our team. The collection and analysis of data at scale is key to Streetbees technology activities as a leading ‘applied AI’ and LLM company - devops practices and scalable infrastructure are fundamental to how we achieve this. We are looking for enthusiasts to contribute to building something truly great in this space.
The Senior Data Engineer will have strong technical knowledge relative to their years of experience but also a keen appreciation for the role devops plays in engineering practices and culture. We are looking for experienced engineers who relish the chance to lead the implementation of core features for an LLM powered data analytics pipeline and play a key role in deciding our data architecture direction. We are more driven by a colleagues contribution and skills than length of service, and as a fast growing start-up we have flexibility in exact skills requirements in order to build the best team. Successful candidates will probably have significant experience in some of these areas; data engineering (e.g. Kafka, Spark, Databricks, Postgres, Airflow etc) and container engineering (e.g. ECS, Docker, Kubernetes etc).
This is a fully hands on position. Experience with Python, Spark and SQL will be essential while experience with AWS, CircleCI (or other automated build platforms), Terraform, Ansible and Datadog are nice to have. Engineers who have a self-motivated interest in developing devops practices and supporting the wider engineering community at Streetbees will find a collaborative environment and a supportive blameless culture.

REQUIREMENTS

• At least 5 year’s experience as a python engineer
• Knowledge of Databricks, Spark, Athena, Delta Lake
• Experience managing relational databases (we use Aurora Postgres)
• Experienced with the AWS cloud platform
• Experience with containers and scheduling tools (we use Docker on ECS)
• Experience working with CI/CD pipeline tools such as CircleCI, Jenkins etc

• Hands-on design, development and support of all python data infrastructure and services in collaboration with the rest of the engineering team. Active pull request participation, branch management, code reviews etc
• Mentoring the development and progression of the Data Engineers
• Coordinate the data engineering workflows in sprint planning with the Head of AI and Product Owners
• Taking an active role in technology decisions relating to our data strategy and architecture
• Developing and maintaining data pipelines for our Lakehouse, LLM request platform, and data operations workflows
• Working closely with the Operations and BI teams to support their activities interacting with the LLM and Lakehouse services
• Investigating bugs and performance issues and providing support to the product engineering teams
• Working closely with full stack developers and product managers to ensure that data infrastructure requirements are implemented correctly and efficiently
• Sharing knowledge within the engineering and operations teams using presentations, demos and pair programming
• Working on production deployment of AI solutions including traditional and LLM models in self hosted and 3rd party provisioned environments
• Keeping up to date with the latest evolutions in engineering best practice and technologies