Senior Data Engineer

Overview We are looking for a Senior Data Engineer to support a long-term engagement focused on simulation data processing for Autonomous Vehicle (AV) development. This role sits at the intersection of large-scale data engineering, real-world sensor analysis, and safety-critical automotive systems. You will work with high-volume multimodal sensor data collected from a test AV fleet and help transform raw inputs into simulation-ready datasets used to develop and validate advanced autonomous driving features. The work directly contributes to next-generation vehicle safety in collaboration with a leading global automotive OEM. What You’ll Work On You’ll handle complex real-world driving data and scenarios such as: • Obstacle detection • Path planning • Complex traffic environments (e.g., tunnels, unusual vehicles, temporary network issues) • Edge cases critical for safe autonomous driving Sensor inputs include 8–12 cameras, LiDAR, and radar, generating up to ~1TB of data per hour. Key Responsibilities • Analyze large-scale real-world sensor datasets to identify edge cases (e.g., hard braking, close-proximity vehicles, unusual road behavior) • Design and write advanced SQL, Python, and Spark/PySpark queries for data filtering, transformation, and preparation • Work with internal platforms for data search, labeling, and auto-labeling workflows • Process structured and semi-structured data, including object detection and perception outputs • Select and prepare relevant data for AV simulation environments and ML pipelines • Contribute to improvements in data discovery and curation processes • Build and maintain data mining scripts and ETL pipelines • Develop internal tools to enhance analytics capabilities and streamline engineering workflows • Collaborate closely with engineers and researchers to support development and validation of safety-critical AV features Requirements • 4+ years of experience in Data Engineering or similar role • Strong software engineering mindset (not a traditional DBA profile) • Advanced SQL (ability to write complex queries) • Advanced Python • Advanced Spark / PySpark • Hands-on experience with Databricks • Experience working with large-scale or complex datasets • Experience in advanced data analytics, including time series analysis • Understanding of ML workflows (data preparation for training/validation) • Solid understanding of data pipelines and distributed data processing • Availability for daily 1-hour overlap with US team (around 6 PM CST) Nice to Have • Experience in the Autonomous Vehicle (AV) or ADAS domain • Exposure to sensor data (camera, LiDAR, radar) • Understanding of real-world driving edge cases • University degree in Computer Science or a related technical field