Company Overview: Atreides helps organizations transform large and complex multi-modal datasets into information-rich, geo-spatial data subscriptions that can be used across a wide spectrum of use cases. Currently, Atreides focuses on providing defence intelligence professionals with high-fidelity data solutions that enable these customers to derive insights in minutes. The company is a fast-moving, fast-growing, high-performance startup.
We value a diverse team and believe inclusion performs better. We believe that trust + autonomy = better results and more job satisfaction. Since we are building something new and helping unlock the power of massive-scale data to make the world safer, stronger, and more prosperous, a mission-driven mindset and entrepreneurial spirit are a must.
Team Overview: We are a passionate team of technologists, data scientists, and analysts with backgrounds in operational intelligence, law enforcement, large multinationals, and cybersecurity operations. We obsess about designing products that will change the way global companies, governments and nonprofits protect themselves from external threats and global adversaries.
Position Overview: We’re looking for a skilled Data Engineer to work on-site with a key customer, specialising in designing, implementing, and optimising data pipelines and storage solutions to manage our extensive datasets, ranging from hundreds of terabytes to petabytes. You will play a key role in ensuring data integrity, accessibility, and security while working closely with customer and partner teams, while supported by the wider Atreides team.
This role is hybrid, based one day per week on-site with an Atreides client based in Cambridgeshire, UK.

DESIRED SKILLS, QUALIFICATIONS, EXPERIENCE:

• 5+ years of experience in data engineering or related roles
• Proficiency in Python, Java, Golang, Spark, Azure, Synapse Analytics, SQL, Datalake Technology, PySpark, Python, Azure Functions, Azure Data Factory Nice to have: Scala.
• Familiarity with Kepler geospatial visualisation tool, or other similar tools, and experience working with geospatial data would also be an advantage
• Knowledge of a wide range of data types and knowledge of integrating different schemas into a single unified schema.
• Strong understanding of data optimization, storage solutions, and data retention policies
• Familiarity with Azure infrastructure and site reliability engineering principles. Azure Data Eng/DevOps Associate or Professional Azure certificates are a plus.
• Knowledge of big data tools and environments, having previously worked with at least 100TB of data at production scale
• Familiarity with geospatial data formats such as Parquet/GeoParquet, GeoJSON, and Shapefiles is a plus.
• Experience with data orchestration tools such as Airflow, Prefect, Dagster, or Temporal, would also be a plus.
• Excellent problem-solving skills and the ability to think quickly in a high-performance environment
• Effective communication skills to convey technical concepts to both technical and non-technical stakeholders
• Ability to work in an Agile team and be self-motivating
• Ability to work independently on a customer-facing project, with minimal supervision
• Ability to travel to the customer site (near Huntingdon, Cambridgeshire), currently once a week, this could be reduced with agreement with the customer
• SC clearance

• Design, develop, and maintain scalable data pipelines, including from external API’s or datasets and fuse them with other data sources.
• Create geospatial data products to be used with visualision tools to facilitate understanding with Analysts and End Users
• Develop and enforce data retention policies and ensure compliance with data governance standards
• Work with a wide range of data types, applying knowledge of integrating different schemas into a single unified schema
• Collaborate with infrastructure engineers to integrate data processes with Azure infrastructure and automation tools
• Ensure data security and implement necessary measures to protect sensitive information
• Monitor and troubleshoot data pipelines to ensure high availability and performance
• Document data engineering processes and maintain comprehensive documentation