Data Engineer

Data Engineer For our client in the pharmaceutical industry, we are looking for an experienced Data Engineer to develop well‑documented pipelines that transform clinical data into curated datasets based on standards such as CDISC SDTM. The role requires close collaboration with clinical experts, data scientists, and engineering teams to enhance and expand existing Databricks medallion architectures. The position also involves integrating Real‑World Data from various registries, using both rule‑based and automated mapping approaches, including OMOP interoperability. Work mode: 100% remote Tasks • Design, build and maintain production ETL pipelines in Databricks/Delta Lake to ingest RWD (registries, claims, EHR extracts) and transform into standard models. • Implement harmonisation workflows to map incoming RWD to OMOP and to the internal CDISC SDTM canonical model; handle vocabulary mapping, units normalization and provenance. • Extend the medallion architecture (bronze/silver/gold) patterns with robust validation, lineage, partitioning and performance tuning. • Develop configurable, input‑driven transformation frameworks so clinical experts can drive mapping rules via config files and catalogs. • Integrate AI/automation components (e.g., model‑assisted mapping, NLP for free text) with human‑in‑the‑loop review and confidence scoring. • Establish testing, CI/CD, monitoring and alerting for ETL jobs and automations; ensure reproducibility, versioning and governance. • Collaborate with clinical data scientists, data stewards and stakeholders to define requirements, data contracts and success metrics. Requirements • Proven experience designing and implementing ETL pipelines in Databricks/Spark and Delta Lake. • Strong knowledge of OMOP CDM and experience mapping datasets to OMOP; familiarity with CDISC SDTM is a plus. • Expertise in data modelling, partitioning, performance tuning, and best practices for large clinical/RWD datasets. • Experience with vocabulary services and terminology mapping (OHDSI/Athena, UMLS, or similar). • Experience integrating AI/NLP components into data pipelines (entity extraction, mapping suggestions) is desirable. • Familiarity with testing frameworks for data (Great Expectations, Deequ), CI/CD, infrastructure as code, and orchestration tools (Databricks Jobs, Airflow). • Good communication skills and experience working with domain experts to capture requirements. • Fluent English  Nice to have • Prior experience in pharma or clinical research environments. • Knowledge of data governance, privacy regulations and secure handling of patient data. • Experience with Unity Catalog, Databricks Delta Sharing, and cloud infrastructure (Azure/AWS). We offer • Ongoing support from a dedicated agent, taking care of your project continuity, client contact, necessary formalities, work comfort and development,  • Consultant Development Program – advice on growth planning based on the latest trends and market needs in IT, including consultations with agents and growth mentors,  • Access to 7N Learning & Development – a development and educational platform with webinars, a library of articles and industry reports, and regular invitations to one-time and recurring development events – technical, business, and lifestyle,  • Spectacular integration events, both for you (e.g., annual Kick-Off trip, Christmas parties, or Summer Olympics sports events) and for your loved ones (e.g., family picnics, movie premieres),  • Professional development not only during the project – you can get involved in knowledge transfer to others within the 7N Services offering directed at 7N clients,  • Relationships and access to the knowledge of the most experienced IT experts in the market – the average professional tenure of our consultants in Poland is over 10 years,  • A complete benefits package, including funding for medical care, life insurance, sports cards for you and your loved ones, as well as discounts in stores in Poland and abroad.  About 7N Constantly searching for projects, difficult rate negotiations, lack of development support – sounds familiar? At 7N, you gain not only stability of contracts but also the personal involvement of a dedicated agent who ensures your professional comfort and continuous access to development initiatives.  Our mission is to provide stable and rewarding collaborations that drive your success as an IT expert and the success of our clients. We build long-lasting relationships based on Scandinavian values and 30 years of experience creating IT solutions for over 200 organizations.