Databricks Data Engineer

Responsibilities: • Developing robust batch and Structured Streaming workflows. • Partnering with BI and Analytics teams on data modeling tasks. • Managing and refining data structures (partitioning, Delta Tables, tuning). • Handling large-scale data processing using PySpark and Spark SQL. • Integrating various sources including APIs, databases, and file systems. • Delivering ETL/ELT pipelines within the Databricks environment. • Overseeing cost efficiency and performance of Databricks clusters. • Applying Lakehouse and Delta Lake architectural solutions. • Managing CI/CD pipelines across DEV, TEST, and PROD stages. • Ensuring high standards through data quality tests and monitoring. Requirements: • 4+ years of professional background in Data Engineering. • Advanced knowledge of Spark SQL, PySpark, and Apache Spark. • Expertise in Delta Lake and Lakehouse frameworks. • Practical, hands-on experience with the Databricks platform. • Proven track record in ETL/ELT design and implementation. • Strong proficiency in SQL and version control (Git). • Background in processing and managing massive datasets. • Familiarity with cloud providers (AWS, Azure, or GCP). • Understanding of Data Quality and Governance principles.