Senior QA Engineer

We are seeking a Senior QA Engineer to ensure the stability, accuracy, and reliability of our Databricks-based data platforms. This role will primarily focus on validating and monitoring data pipelines, ETL processes, and backend data workflows within our Data Lake and Big Data environments. You will be responsible for implementing automated data quality checks, validating Databricks jobs regularly, and ensuring data integrity across the Data Market Hub platform. The position involves hands-on testing of data transformations using Python, PySpark, and SQL, executing pipeline validation checks, and supporting release cycles through structured QA processes. You will work closely with data engineers and platform teams to proactively identify issues, validate data accuracy, and maintain consistent quality standards across all data assets. Additionally, you will contribute to test reporting, dashboarding, and continuous improvement of QA practices within the data platform ecosystem. - Automate data quality checks for Databricks (DBRX), Data Lake, and Big Data environments. - Create and maintain test plans for the Data Market Hub platform. - Develop and maintain Data Quality and Data Expectation checks within the Market Data Hub Platform using Python/PySpark - Participate in requirements refinement and specification. - Use SQL to validate data integrity and support efficient data processing workflows. - Collaborate in the incident management process to ensure swift issue resolution. - Test Data Management: Use SQL to manage test data, retrieve relevant test information, and store test execution results in a centralized database for historical analysis and reporting. - Dashboard Design: Create PowerBI dashboards to provide insights into test results, code coverage, and performance trends across projects and test levels.