Senior Databricks Engineer (Architect)

Databricks Architect Project: Legacy Migration from SAS to Databricks Key Responsibilities • Migration Strategy: Lead the end-to-end migration of complex data pipelines from SAS to Databricks. • Architecture Design: Design scalable PySpark/Delta Lake solutions to replace legacy SAS Base/Macro code. • Code Transformation: Deconstruct existing SAS logic and re-engineer it into optimized, modern ETL/ELT workflows. • Performance Tuning: Optimize Spark jobs and SQL queries within Databricks for maximum cost-efficiency and speed. • Stakeholder Guidance: Act as a technical bridge between legacy data owners and the modern cloud engineering team. Technical Requirements • Databricks Expert: Extensive experience with PySpark, Delta Lake, and Unity Catalog. • SAS Specialist: Deep understanding of SAS datasets, procedures, and macros (ability to read and translate legacy code). • Senior+ Experience: Proven track record in architectural roles or leading large-scale data migrations. • Cloud Proficiency: Hands-on experience with Azure (preferred) or AWS data services. • Software Craftsmanship: Proficiency in Python and SQL at an advanced level.