Data Engineer

Job Description: Data Engineer, LDWH Migration Programme
Level: Senior / Mid-Senior

Main Responsibilities:
• Develop and maintain data pipelines using Azure Databricks.
• Implement ETL processes, ensuring data quality and integrity.
• Read and comprehend complex SQL stored procedures.
• Translate architecture direction into engineering delivery.
• Conduct data profiling and quality assessments.
• Define acceptance criteria for data transformation.
• Collaborate with the programme team to support overall migration efforts.
• Participate in CI/CD processes for efficient deployment.
Key Requirements:
• Senior hands-on experience with Azure Databricks (jobs, clusters, Unity Catalog, SQL warehouses).
• Expertise in PySpark / Python for transformation logic and ETL pipelines.
• Solid working knowledge of Delta Lake (schema evolution, MERGE, time travel).
• Advanced SQL skills (complex queries, window functions); ability to comprehend legacy ETL.
• Familiarity with Databricks Asset Bundles (DAB) and CI/CD practices (Azure DevOps, Git).
• Working knowledge of Azure Data Lake Storage Gen2.
• Comfortable understanding legacy ETL processes and source data profiling.
• Experience with data reconciliation (defining criteria, interpreting discrepancies).
• Ability to articulate requirements for junior engineers.
• Fundamental knowledge of Power BI / Tableau.
Nice to Have:
• Experience in platform engineering and configuration refactoring using YAML.
• Ability to maintain environment governance across DEV, QAS, and PRD.
• Skills in source feasibility analysis for new data sources.
• Capability in transformation development across various data zones using PySpark and Delta Lake.
• Familiar with implementing acceptance criteria and conducting quality checks.
Other Details:
This position is part of a complex and dynamic migration programme and is pivotal for ensuring the successful transition of data systems. Remote engagement options are available, and collaboration with an internal team is essential.