Senior Data Engineer (with Backend Experience)

O projekcie: Get to know us better CodiLime is a software and network engineering industry expert and the first-choice service partner for top global networking hardware providers, software providers and telecoms. We create proofs-of-concept, help our clients build new products, nurture existing ones and provide services in production environments. Our clients include both tech startups and big players in various industries and geographic locations (US, Japan, Israel, Europe). While no longer a startup - we have 250+ people on board and have been operating since 2011 we’ve kept our people-oriented culture. Our values are simple: - Act to deliver. - Disrupt to grow. - Team up to win. The project and the team You will work on a large-scale data platform built for a global consulting firm, supporting M&A (Mergers and Acquisitions) analysis. The project consists of: - A cloud-based platform for data visualization (APIs and services) - A large, integrated dataset (300M+ company records from 10+ sources) Your primary focus will be on data engineering (approx. 80%), including building and improving data pipelines and dataset quality. Additionally, you will contribute to backend and platform development (approx. 20%), including features, APIs, and integration with the data layer. Technology stack: - Languages: Python, SQL - Data: Snowflake, dbt - Workflow Orchestration: Apache Airflow (complex DAGs) - Data Processing: Apache Spark - Platform - Cloud: - AWS (EKS, S3, Lambda, ECR, EMR, Opensearch) - Platform - Azure (AKS, Blob Storage, Azure Functions, ACR, Databricks, AI Search) - Dataset - API: API Gateway, FastAPI (REST, async) - CI/CD: GitHub Actions - Data Scale: 300M+ records across multiple sources - Future Direction: - Agentic AI systems - LangChain and cloud-native AI libraries and frameworks - Azure OpenAI integration What else you should know: - Team: Data Architecture Lead, Data Engineers, DataOps Engineers, Frontend Engineer, Backend Engineer, Product Owner - Distributed team across Europe and India - Agile, collaborative, proactive and impact-driven environment We work on multiple projects at the same time, so we may suggest a different project if it better matches your experience and profile. Your role This is a hybrid role combining Data Engineering and Backend Engineering, with a stronger focus on the data layer. More reasons to join us - Flexible working hours and approach to work: fully remotely, in the office or hybrid - Professional growth supported by internal training sessions and a training budget - Solid onboarding with a hands-on approach to give you an easy start - A great atmosphere among professionals who are passionate about their work Wymagania: Technology stack: - Languages: Python, SQL - Data: Snowflake, dbt - Workflow Orchestration: Apache Airflow (complex DAGs) - Data Processing: Apache Spark - Platform - Cloud: - AWS (EKS, S3, Lambda, ECR, EMR, Opensearch) - Platform - Azure (AKS, Blob Storage, Azure Functions, ACR, Databricks, AI Search) - Dataset - API: API Gateway, FastAPI (REST, async) - CI/CD: GitHub Actions - Data Scale: 300M+ records across multiple sources - Future Direction: - Agentic AI systems - LangChain and cloud-native AI libraries and frameworks - Azure OpenAI integration What else you should know: - Team: Data Architecture Lead, Data Engineers, DataOps Engineers, Frontend Engineer, Backend Engineer, Product Owner - Distributed team across Europe and India - Agile, collaborative, proactive and impact-driven environment As a Senior Data Engineer, you must meet the following criteria: - Strong experience with Snowflake and dbt - Strong Python skills - Ability to write clean, scalable, and testable code (including unit tests) - Solid understanding and applying object-oriented programming (OOP) - Experience with version control systems: Git - Experience with AWS or Azure - Good knowledge of English (minimum C1 level) Beyond the criteria above, we would appreciate the nice-to-haves: - Experience with Apache Spark (preferably Databricks) - Experience with Azure AI Search or AWS OpenSearch - Experience designing ETL/ELT pipelines - Familiarity with LLMs, Azure OpenAI, or agentic AI systems Codzienne zadania: - Build and maintain scalable ETL/data pipelines (Python, SQL, dbt, Snowflake, Airflow) - Design and optimize data models, transformations and workflows - Ensure data quality, consistency, and performance - Integrate multiple data sources into a unified dataset - Contribute to backend development (APIs, data access layer, backend-frontend contracts) - Support platform integrations and cloud-based solutions - Write clean, testable code and participate in code reviews - Document data processes and system architecture