Senior Data Engineer at Spyrosoft

Buenos Aires, , Argentina -

Full Time

Start Date

Immediate

Expiry Date

11 Mar, 26

Salary

0.0

Posted On

11 Dec, 25

Experience

5 year(s) or above

Remote Job

Yes

Telecommute

Yes

Sponsor Visa

Skills

SQL, Python, Spark, PySpark, Databricks, ML Workflows, Data Mining, ETL Processes, Data Filtering, Data Transformation, Data Curation, Analytics, Simulation Data Processing, Object Detection, Event-Detection Rules, Complex Query Logic

Industry

IT Services and IT Consulting

Description

We are looking for a highly analytical data engineer who can design and implement complex SQL and PySpark logic over massive, high-frequency time-series data from real-world vehicle sensor recordings. This role requires the ability to craft intricate temporal and spatial query logic—including multi-table joins, windowing across time, and event-detection rules—to extract meaningful driving behaviors from a rich 3D world-model (obstacles, lanes, traffic signs, etc.). The ideal candidate excels at breaking down complex behavioral patterns into precise, scalable query logic and optimizing these queries to run efficiently on large datasets. We are also looking for someone who can work independently. Project Description This long-term engagement supports simulation data processing for Autonomous Vehicle (AV) development. Key scenarios include obstacle detection, path planning, and complex traffic situations (e.g., tunnels, unusual vehicles, or temporary network issues). The candidate will work with high-volume sensor data from a test AV fleet (8–12 cameras, LiDAR, radar), generating up to ~1TB/hour. Familiarity with the AV domain and real-world edge cases will be valuable. The team collaborates closely with other team members to develop and validate safety-critical AV features. The scope includes full-cycle data curation—from raw sensor input to simulation-ready datasets—and close cooperation with engineers and researchers. Key Responsibilities: • Analyze real-world sensor data to identify edge cases (e.g., hard braking, nearby vehicles) • Create advanced SQL, Python, and Spark/PySpark queries for data filtering and transformation • Work with internal tools for data search and auto-labeling workflows • Process structured/semi-structured data (e.g., object detection output) • Identify relevant data for AV simulations and ML pipelines • Suggest and validate improvements in data discovery processes • Build and maintain data mining scripts and ETL processes • Develop tools to enhance analytics and streamline workflows Requirements • SQL (advanced) • Python (advanced) • Spark / PySpark (advanced) • Hands-on experience with Databricks • Understanding of ML workflows • University degree in Computer Science (nice to have) • Over 4 years of commercial experience

Responsibilities

The candidate will analyze real-world sensor data to identify edge cases and create advanced SQL, Python, and Spark/PySpark queries for data filtering and transformation. They will also build and maintain data mining scripts and ETL processes while developing tools to enhance analytics and streamline workflows.