Senior Data Engineer

O projekcie: We are looking for a Senior Data Engineer to support a long-term engagement focused on simulation data processing for Autonomous Vehicle (AV) development. This role sits at the intersection of large-scale data engineering, real-world sensor analysis, and safety-critical automotive systems. You will work with high-volume multimodal sensor data collected from a test AV fleet and help transform raw inputs into simulation-ready datasets used to develop and validate advanced autonomous driving features. The work directly contributes to next-generation vehicle safety in collaboration with a leading global automotive OEM. What You’ll Work On You’ll handle complex real-world driving data and scenarios such as: - Obstacle detection - Path planning - Complex traffic environments (e.g., tunnels, unusual vehicles, temporary network issues) - Edge cases critical for safe autonomous driving Sensor inputs include 8–12 cameras, LiDAR, and radar, generating up to ~1TB of data per hour. About Spyrosoft Spyrosoft is an authentic, cutting-edge software engineering company, established in 2016. In 2021 and 2022, we were among the fastest growing technology companies in Europe, according to the Financial Times. We were founded by a group of tech experts with established backgrounds in software engineering, who created an ‘engineer-to-engineer’ workplace, powered by enthusiasm, fairness and authentic relationships. Having a unique offering, which bridge the gap between technology and business, we specialise in technology solutions for industry 4.0, automotive, geospatial, healthcare & life sciences, employee experience & education and financial services industries. Wymagania: - 4+ years of experience in Data Engineering or similar role - Strong software engineering mindset (not a traditional DBA profile) - Advanced SQL (ability to write complex queries) - Advanced Python - Advanced Spark / PySpark - Hands-on experience with Databricks - Experience working with large-scale or complex datasets - Experience in advanced data analytics, including time series analysis - Understanding of ML workflows (data preparation for training/validation) - Solid understanding of data pipelines and distributed data processing - Availability for daily 1-hour overlap with US team (around 6 PM CST) Nice to Have - Experience in the Autonomous Vehicle (AV) or ADAS domain - Exposure to sensor data (camera, LiDAR, radar) - Understanding of real-world driving edge cases - University degree in Computer Science or a related technical field Codzienne zadania: - Analyze large-scale real-world sensor datasets to identify edge cases (e.g., hard braking, close-proximity vehicles, unusual road behavior) - Design and write advanced SQL, Python, and Spark/PySpark queries for data filtering, transformation, and preparation - Work with internal platforms for data search, labeling, and auto-labeling workflows - Process structured and semi-structured data, including object detection and perception outputs - Select and prepare relevant data for AV simulation environments and ML pipelines - Contribute to improvements in data discovery and curation processes - Build and maintain data mining scripts and ETL pipelines - Develop internal tools to enhance analytics capabilities and streamline engineering workflows - Collaborate closely with engineers and researchers to support development and validation of safety-critical AV features