Senior Databricks Engineer (Architect)

O projekcie: Databricks Architect Project: Legacy Migration from SAS to Databricks Wymagania: - Databricks Expert: Extensive experience with PySpark, Delta Lake, and Unity Catalog. - SAS Specialist: Deep understanding of SAS datasets, procedures, and macros (ability to read and translate legacy code). - Senior+ Experience: Proven track record in architectural roles or leading large-scale data migrations. - Cloud Proficiency: Hands-on experience with Azure (preferred) or AWS data services. - Software Craftsmanship: Proficiency in Python and SQL at an advanced level. Codzienne zadania: - Migration Strategy: Lead the end-to-end migration of complex data pipelines from SAS to Databricks. - Architecture Design: Design scalable PySpark/Delta Lake solutions to replace legacy SAS Base/Macro code. - Code Transformation: Deconstruct existing SAS logic and re-engineer it into optimized, modern ETL/ELT workflows. - Performance Tuning: Optimize Spark jobs and SQL queries within Databricks for maximum cost-efficiency and speed. - Stakeholder Guidance: Act as a technical bridge between legacy data owners and the modern cloud engineering team.