Senior Data Engineer

Introduction & Summary We are seeking a Senior Data Engineer to build data systems and applications that bolster decision-making processes. This role focuses on developing and maintaining data pipelines in public clouds like AWS while participating in data migrations from on-premise environments. Ideal candidates will possess advanced technical expertise and strong communication skills, facilitating team collaboration in an Agile setup. Main Responsibilities The Senior Data Engineer will be responsible for: • Building distributed and highly parallelized big data processing pipelines to handle massive amounts of structured and unstructured data in near real-time. • Utilizing Spark to enrich and transform corporate data, enabling effective searching, visualization, and advanced analytics. • Collaborating with analysts and business stakeholders to develop analytics models. • Ensuring continuous delivery on Hadoop and other Big Data platforms. • Automating processes to enhance repeatability and reliability. • Cooperating closely with the QA team. Key Requirements • Minimum 5 years of experience in Spark & Scala. • Minimum 7 years of experience in Python. • Minimum 5 years of experience in Linux. • Experience with distributed data processing engines like Spark. • Experience with Hadoop and related technologies (Hive, Oozie, MapReduce, etc.). • Strong/advanced SQL skills. • Proven experience in creating data flows. • Expertise in BitBucket and GIT. • AWS certification or experience is a plus. • Unit testing experience (JUnit 5, Mockito, Spark testing). • Knowledge of code versioning strategy and branching strategy. • Advanced English language skills. Preferable location Trójmiasto / Warszawa or Łodź Hybrid work possible with 1 day/week office presence + once/twice per year physical presence in Gdynia/Gdańsk for quarterly PI Planning (2 days) may be required